Overview

Workflow Report

This is a HTML-ification of Workflow state (workflow.state) and the associated files in the workflow output directory. Below is the doc section of the workflow (.workflow file). 

Produce, transform and plot fio-output
======================================

This workflow assumes that the 'prep.workflow' has completed.
The comparison and plots consists of

  * bandwidth in KiB / seconds
  * latency is mean and in nano-seconds
  * IOPS are 'raw' / base-unit

The above in a bar-plot at iodepth=1, and a line-plot as a function of iodepth.

See Artifacts for the generated plots
The workflow started on 26-10-2022, 02:11:00, status on the steps is provided below.

If a workflow step (worklet / run) or test produced any artifacts, then they are listed here.

This is the configuration (.config file) with any variable-placeholders such as {{ local.env.HOME }} filled out. 

devices:
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme0n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme1n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme2n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme3n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme4n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme5n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme6n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme7n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng0n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng1n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng2n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng3n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng4n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng5n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng6n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng7n1
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:41:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:42:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:43:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:44:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:45:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:46:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:47:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:48:00.0'
fio:
  bin: /opt/fio/bin/fio
  build:
    prefix: /opt/fio
  engines:
    io_uring:
      type: builtin
    io_uring_cmd:
      type: builtin
    libaio:
      type: builtin
    spdk_bdev:
      path: /root/git/spdk/build/fio/spdk_bdev
      type: external_preload
    spdk_nvme:
      path: /root/git/spdk/build/fio/spdk_nvme
      type: external_preload
    xnvme:
      path: /usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so
      type: external_dynamic
  repository:
    path: /root/git/fio
    remote: https://github.com/axboe/fio.git
    tag: fio-3.32
liburing:
  repository:
    path: /root/git/liburing
    remote: https://github.com/axboe/liburing.git
    tag: liburing-2.2
os:
  name: debian
  version: bullseye
spdk:
  build:
    prefix: /opt/spdk
  repository:
    branch: for_bdev_xnvme_v6
    path: /root/git/spdk
    remote: https://github.com/safl/spdk.git
transport:
  ssh:
    hostname: fabio
    port: 22
    username: root
xnvme:
  build:
    type: debug
  repository:
    path: /root/git/xnvme
    remote: https://github.com/OpenMPDK/xNVMe.git
    tag: v0.6.0

Steps

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Run fio...

{'description': '\n    Run fio...\n',
 'extras': {'runlog': {'cmd_01': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.output'),
                                  'state': {'begin': 1666743060.3786855,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8657588958740234,
                                            'end': 1666743061.2444444,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_01.state')},
                       'cmd_02': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '171834.77 IOPS      83.90 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '171834.77 IOPS      83.90 MiB/s\n'
                                            '[2022-10-26 04:09:12.283172] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:09:12.283242] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:09:12.283253] bdevperf '
                                            '[2022-10-26 04:09:12.283263] '
                                            '--no-shconf [2022-10-26 '
                                            '04:09:12.283271] -l 0 [2022-10-26 '
                                            '04:09:12.283280] --huge-unlink '
                                            '[2022-10-26 04:09:12.283289] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:09:12.283296] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:09:12.283304] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:09:12.283312] --iova-mode=pa '
                                            '[2022-10-26 04:09:12.283320] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:09:12.283328] '
                                            '--match-allocations [2022-10-26 '
                                            '04:09:12.283335] '
                                            '--file-prefix=spdk_pid137152 '
                                            '[2022-10-26 04:09:12.283344] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:09:12.421601] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:09:12.473509] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:09:12.474184] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.output'),
                                  'state': {'begin': 1666743062.2779996,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.123723030090332,
                                            'end': 1666743073.4017227,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_02.state')},
                       'cmd_03': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.output'),
                                  'state': {'begin': 1666743073.4047213,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7753987312316895,
                                            'end': 1666743074.18012,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_03.state')},
                       'cmd_04': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '172633.05 IOPS      84.29 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '172633.05 IOPS      84.29 MiB/s\n'
                                            '[2022-10-26 04:09:25.202810] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:09:25.202878] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:09:25.202890] bdevperf '
                                            '[2022-10-26 04:09:25.202898] '
                                            '--no-shconf [2022-10-26 '
                                            '04:09:25.202906] -l 0 [2022-10-26 '
                                            '04:09:25.202914] --huge-unlink '
                                            '[2022-10-26 04:09:25.202921] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:09:25.202930] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:09:25.202937] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:09:25.202944] --iova-mode=pa '
                                            '[2022-10-26 04:09:25.202952] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:09:25.202959] '
                                            '--match-allocations [2022-10-26 '
                                            '04:09:25.202967] '
                                            '--file-prefix=spdk_pid137177 '
                                            '[2022-10-26 04:09:25.202974] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:09:25.341687] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:09:25.389529] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:09:25.390195] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.output'),
                                  'state': {'begin': 1666743075.2015347,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.11318588256836,
                                            'end': 1666743086.3147206,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_04.state')},
                       'cmd_05': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.output'),
                                  'state': {'begin': 1666743086.3168955,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.782501220703125,
                                            'end': 1666743087.0993967,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_05.state')},
                       'cmd_06': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '171552.75 IOPS      83.77 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '171552.75 IOPS      83.77 MiB/s\n'
                                            '[2022-10-26 04:09:38.130730] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:09:38.130800] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:09:38.130811] bdevperf '
                                            '[2022-10-26 04:09:38.130819] '
                                            '--no-shconf [2022-10-26 '
                                            '04:09:38.130828] -l 0 [2022-10-26 '
                                            '04:09:38.130836] --huge-unlink '
                                            '[2022-10-26 04:09:38.130845] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:09:38.130853] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:09:38.130860] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:09:38.130868] --iova-mode=pa '
                                            '[2022-10-26 04:09:38.130876] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:09:38.130885] '
                                            '--match-allocations [2022-10-26 '
                                            '04:09:38.130893] '
                                            '--file-prefix=spdk_pid137202 '
                                            '[2022-10-26 04:09:38.130901] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:09:38.269317] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:09:38.321522] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:09:38.322192] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.output'),
                                  'state': {'begin': 1666743088.124827,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.126126766204834,
                                            'end': 1666743099.2509537,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_06.state')},
                       'cmd_07': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.output'),
                                  'state': {'begin': 1666743099.2537508,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7822954654693604,
                                            'end': 1666743100.0360463,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_07.state')},
                       'cmd_08': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '129651.27 IOPS      63.31 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '129651.27 IOPS      63.31 MiB/s\n'
                                            '[2022-10-26 04:09:51.072421] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:09:51.072495] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:09:51.072506] bdevperf '
                                            '[2022-10-26 04:09:51.072515] '
                                            '--no-shconf [2022-10-26 '
                                            '04:09:51.072524] -l 0 [2022-10-26 '
                                            '04:09:51.072532] --huge-unlink '
                                            '[2022-10-26 04:09:51.072541] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:09:51.072549] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:09:51.072557] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:09:51.072566] --iova-mode=pa '
                                            '[2022-10-26 04:09:51.072574] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:09:51.072583] '
                                            '--match-allocations [2022-10-26 '
                                            '04:09:51.072591] '
                                            '--file-prefix=spdk_pid137229 '
                                            '[2022-10-26 04:09:51.072600] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:09:51.210930] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:09:51.257547] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:09:51.258212] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.output'),
                                  'state': {'begin': 1666743101.0615962,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.104482889175415,
                                            'end': 1666743112.166079,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_08.state')},
                       'cmd_09': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.output'),
                                  'state': {'begin': 1666743112.16912,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7827787399291992,
                                            'end': 1666743112.9518988,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_09.state')},
                       'cmd_10': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '129366.45 IOPS      63.17 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '129366.45 IOPS      63.17 MiB/s\n'
                                            '[2022-10-26 04:10:03.980551] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:10:03.980620] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:10:03.980631] bdevperf '
                                            '[2022-10-26 04:10:03.980640] '
                                            '--no-shconf [2022-10-26 '
                                            '04:10:03.980648] -l 0 [2022-10-26 '
                                            '04:10:03.980655] --huge-unlink '
                                            '[2022-10-26 04:10:03.980663] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:10:03.980671] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:10:03.980678] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:10:03.980686] --iova-mode=pa '
                                            '[2022-10-26 04:10:03.980693] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:10:03.980700] '
                                            '--match-allocations [2022-10-26 '
                                            '04:10:03.980708] '
                                            '--file-prefix=spdk_pid137254 '
                                            '[2022-10-26 04:10:03.980716] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:10:04.119553] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:10:04.161522] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:10:04.162184] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.output'),
                                  'state': {'begin': 1666743113.9712324,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.115413665771484,
                                            'end': 1666743125.086646,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_10.state')},
                       'cmd_100': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '519280.46 IOPS     253.55 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '519280.46 IOPS     253.55 MiB/s\n'
                                             '[2022-10-26 04:19:47.360269] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:19:47.360339] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:19:47.360350] bdevperf '
                                             '[2022-10-26 04:19:47.360359] '
                                             '--no-shconf [2022-10-26 '
                                             '04:19:47.360367] -l 0 '
                                             '[2022-10-26 04:19:47.360375] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:19:47.360384] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:19:47.360392] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:19:47.360399] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:19:47.360407] --iova-mode=pa '
                                             '[2022-10-26 04:19:47.360416] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:19:47.360423] '
                                             '--match-allocations [2022-10-26 '
                                             '04:19:47.360431] '
                                             '--file-prefix=spdk_pid138396 '
                                             '[2022-10-26 04:19:47.360440] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:19:47.498685] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:19:47.549512] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:19:47.550170] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.output'),
                                   'state': {'begin': 1666743697.3514316,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_uring_foo.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.117814302444458,
                                             'end': 1666743708.469246,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_100.state')},
                       'cmd_101': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.output'),
                                   'state': {'begin': 1666743708.4720542,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_uring_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7855734825134277,
                                             'end': 1666743709.2576277,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_101.state')},
                       'cmd_102': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '520408.18 IOPS     254.11 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '520408.18 IOPS     254.11 MiB/s\n'
                                             '[2022-10-26 04:20:00.275836] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:20:00.275907] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:20:00.275918] bdevperf '
                                             '[2022-10-26 04:20:00.275926] '
                                             '--no-shconf [2022-10-26 '
                                             '04:20:00.275934] -l 0 '
                                             '[2022-10-26 04:20:00.275942] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:20:00.275949] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:20:00.275957] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:20:00.275964] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:20:00.275972] --iova-mode=pa '
                                             '[2022-10-26 04:20:00.275979] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:20:00.275987] '
                                             '--match-allocations [2022-10-26 '
                                             '04:20:00.275994] '
                                             '--file-prefix=spdk_pid138421 '
                                             '[2022-10-26 04:20:00.276001] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:20:00.414472] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:20:00.453507] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:20:00.454165] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.output'),
                                   'state': {'begin': 1666743710.2809129,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_uring_foo.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.092379808425903,
                                             'end': 1666743721.3732927,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_102.state')},
                       'cmd_103': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.output'),
                                   'state': {'begin': 1666743721.37594,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7914619445800781,
                                             'end': 1666743722.167402,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_103.state')},
                       'cmd_104': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '534233.37 IOPS     260.86 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '534233.37 IOPS     260.86 MiB/s\n'
                                             '[2022-10-26 04:20:13.190404] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:20:13.190474] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:20:13.190484] bdevperf '
                                             '[2022-10-26 04:20:13.190494] '
                                             '--no-shconf [2022-10-26 '
                                             '04:20:13.190502] -l 0 '
                                             '[2022-10-26 04:20:13.190509] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:20:13.190517] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:20:13.190525] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:20:13.190532] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:20:13.190539] --iova-mode=pa '
                                             '[2022-10-26 04:20:13.190546] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:20:13.190554] '
                                             '--match-allocations [2022-10-26 '
                                             '04:20:13.190561] '
                                             '--file-prefix=spdk_pid138447 '
                                             '[2022-10-26 04:20:13.190568] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:20:13.328990] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:20:13.369543] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:20:13.370218] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.output'),
                                   'state': {'begin': 1666743723.1897388,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.102015495300293,
                                             'end': 1666743734.2917542,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_104.state')},
                       'cmd_105': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.output'),
                                   'state': {'begin': 1666743734.2947047,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7894554138183594,
                                             'end': 1666743735.08416,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_105.state')},
                       'cmd_106': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '527043.53 IOPS     257.35 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '527043.53 IOPS     257.35 MiB/s\n'
                                             '[2022-10-26 04:20:26.114969] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:20:26.115039] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:20:26.115050] bdevperf '
                                             '[2022-10-26 04:20:26.115059] '
                                             '--no-shconf [2022-10-26 '
                                             '04:20:26.115067] -l 0 '
                                             '[2022-10-26 04:20:26.115075] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:20:26.115083] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:20:26.115090] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:20:26.115097] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:20:26.115105] --iova-mode=pa '
                                             '[2022-10-26 04:20:26.115123] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:20:26.115131] '
                                             '--match-allocations [2022-10-26 '
                                             '04:20:26.115138] '
                                             '--file-prefix=spdk_pid138472 '
                                             '[2022-10-26 04:20:26.115146] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:20:26.253677] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:20:26.301556] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:20:26.302227] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.output'),
                                   'state': {'begin': 1666743736.121251,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.103857517242432,
                                             'end': 1666743747.2251086,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_106.state')},
                       'cmd_107': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.output'),
                                   'state': {'begin': 1666743747.2278082,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.792818546295166,
                                             'end': 1666743748.0206268,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_107.state')},
                       'cmd_108': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '530831.57 IOPS     259.20 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '530831.57 IOPS     259.20 MiB/s\n'
                                             '[2022-10-26 04:20:39.084853] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:20:39.084921] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:20:39.084931] bdevperf '
                                             '[2022-10-26 04:20:39.084940] '
                                             '--no-shconf [2022-10-26 '
                                             '04:20:39.084949] -l 0 '
                                             '[2022-10-26 04:20:39.084957] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:20:39.084966] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:20:39.084974] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:20:39.084981] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:20:39.084990] --iova-mode=pa '
                                             '[2022-10-26 04:20:39.084998] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:20:39.085006] '
                                             '--match-allocations [2022-10-26 '
                                             '04:20:39.085015] '
                                             '--file-prefix=spdk_pid138497 '
                                             '[2022-10-26 04:20:39.085023] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:20:39.223403] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:20:39.269614] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:20:39.270283] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.output'),
                                   'state': {'begin': 1666743749.092826,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.099281311035156,
                                             'end': 1666743760.1921072,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_108.state')},
                       'cmd_109': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.output'),
                                   'state': {'begin': 1666743760.1946323,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7804458141326904,
                                             'end': 1666743760.975078,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_109.state')},
                       'cmd_11': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.output'),
                                  'state': {'begin': 1666743125.0898678,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7840397357940674,
                                            'end': 1666743125.8739076,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_11.state')},
                       'cmd_110': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '533336.85 IOPS     260.42 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '533336.85 IOPS     260.42 MiB/s\n'
                                             '[2022-10-26 04:20:52.015515] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:20:52.015584] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:20:52.015595] bdevperf '
                                             '[2022-10-26 04:20:52.015604] '
                                             '--no-shconf [2022-10-26 '
                                             '04:20:52.015612] -l 0 '
                                             '[2022-10-26 04:20:52.015620] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:20:52.015628] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:20:52.015636] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:20:52.015643] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:20:52.015650] --iova-mode=pa '
                                             '[2022-10-26 04:20:52.015658] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:20:52.015665] '
                                             '--match-allocations [2022-10-26 '
                                             '04:20:52.015673] '
                                             '--file-prefix=spdk_pid138522 '
                                             '[2022-10-26 04:20:52.015680] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:20:52.154077] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:20:52.193585] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:20:52.194258] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.output'),
                                   'state': {'begin': 1666743762.00147,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.181623220443726,
                                             'end': 1666743773.1830933,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_110.state')},
                       'cmd_111': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.output'),
                                   'state': {'begin': 1666743773.1863654,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7893142700195312,
                                             'end': 1666743773.9756796,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_111.state')},
                       'cmd_112': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '536250.58 IOPS     261.84 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '536250.58 IOPS     261.84 MiB/s\n'
                                             '[2022-10-26 04:21:05.002701] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:21:05.002771] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:21:05.002782] bdevperf '
                                             '[2022-10-26 04:21:05.002791] '
                                             '--no-shconf [2022-10-26 '
                                             '04:21:05.002800] -l 0 '
                                             '[2022-10-26 04:21:05.002808] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:21:05.002816] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:21:05.002823] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:21:05.002831] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:21:05.002846] --iova-mode=pa '
                                             '[2022-10-26 04:21:05.002854] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:21:05.002861] '
                                             '--match-allocations [2022-10-26 '
                                             '04:21:05.002870] '
                                             '--file-prefix=spdk_pid138547 '
                                             '[2022-10-26 04:21:05.002878] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:21:05.141323] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:21:05.185509] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:21:05.186190] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.output'),
                                   'state': {'begin': 1666743775.006602,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.164662837982178,
                                             'end': 1666743786.171265,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_112.state')},
                       'cmd_113': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.output'),
                                   'state': {'begin': 1666743786.1741786,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7843453884124756,
                                             'end': 1666743786.958524,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_113.state')},
                       'cmd_114': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '534521.72 IOPS     261.00 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '534521.72 IOPS     261.00 MiB/s\n'
                                             '[2022-10-26 04:21:17.969960] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:21:17.970030] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:21:17.970041] bdevperf '
                                             '[2022-10-26 04:21:17.970050] '
                                             '--no-shconf [2022-10-26 '
                                             '04:21:17.970058] -l 0 '
                                             '[2022-10-26 04:21:17.970066] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:21:17.970074] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:21:17.970081] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:21:17.970088] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:21:17.970096] --iova-mode=pa '
                                             '[2022-10-26 04:21:17.970103] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:21:17.970111] '
                                             '--match-allocations [2022-10-26 '
                                             '04:21:17.970118] '
                                             '--file-prefix=spdk_pid138572 '
                                             '[2022-10-26 04:21:17.970126] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:21:18.108770] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:21:18.149565] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:21:18.150241] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.output'),
                                   'state': {'begin': 1666743787.9747603,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.15734577178955,
                                             'end': 1666743799.132106,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_114.state')},
                       'cmd_115': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.output'),
                                   'state': {'begin': 1666743799.1346714,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7830836772918701,
                                             'end': 1666743799.9177551,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_115.state')},
                       'cmd_116': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '510627.58 IOPS     249.33 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '510627.58 IOPS     249.33 MiB/s\n'
                                             '[2022-10-26 04:21:30.943091] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:21:30.943175] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:21:30.943186] bdevperf '
                                             '[2022-10-26 04:21:30.943195] '
                                             '--no-shconf [2022-10-26 '
                                             '04:21:30.943204] -l 0 '
                                             '[2022-10-26 04:21:30.943211] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:21:30.943220] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:21:30.943227] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:21:30.943235] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:21:30.943243] --iova-mode=pa '
                                             '[2022-10-26 04:21:30.943251] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:21:30.943259] '
                                             '--match-allocations [2022-10-26 '
                                             '04:21:30.943267] '
                                             '--file-prefix=spdk_pid138597 '
                                             '[2022-10-26 04:21:30.943275] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:21:31.081602] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:21:31.121496] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:21:31.122161] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.output'),
                                   'state': {'begin': 1666743800.9505537,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.176557779312134,
                                             'end': 1666743812.1271114,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_116.state')},
                       'cmd_117': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.output'),
                                   'state': {'begin': 1666743812.1298392,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7924110889434814,
                                             'end': 1666743812.9222503,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_117.state')},
                       'cmd_118': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '507405.68 IOPS     247.76 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '507405.68 IOPS     247.76 MiB/s\n'
                                             '[2022-10-26 04:21:43.951587] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:21:43.951660] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:21:43.951671] bdevperf '
                                             '[2022-10-26 04:21:43.951680] '
                                             '--no-shconf [2022-10-26 '
                                             '04:21:43.951688] -l 0 '
                                             '[2022-10-26 04:21:43.951696] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:21:43.951704] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:21:43.951712] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:21:43.951719] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:21:43.951726] --iova-mode=pa '
                                             '[2022-10-26 04:21:43.951734] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:21:43.951741] '
                                             '--match-allocations [2022-10-26 '
                                             '04:21:43.951749] '
                                             '--file-prefix=spdk_pid138623 '
                                             '[2022-10-26 04:21:43.951756] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:21:44.090207] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:21:44.129577] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:21:44.130245] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.output'),
                                   'state': {'begin': 1666743813.9570987,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.194416761398315,
                                             'end': 1666743825.1515155,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_118.state')},
                       'cmd_119': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.output'),
                                   'state': {'begin': 1666743825.1540108,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7881972789764404,
                                             'end': 1666743825.942208,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_119.state')},
                       'cmd_12': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '129066.85 IOPS      63.02 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '129066.85 IOPS      63.02 MiB/s\n'
                                            '[2022-10-26 04:10:16.900959] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:10:16.901031] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:10:16.901042] bdevperf '
                                            '[2022-10-26 04:10:16.901050] '
                                            '--no-shconf [2022-10-26 '
                                            '04:10:16.901058] -l 0 [2022-10-26 '
                                            '04:10:16.901066] --huge-unlink '
                                            '[2022-10-26 04:10:16.901074] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:10:16.901081] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:10:16.901088] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:10:16.901096] --iova-mode=pa '
                                            '[2022-10-26 04:10:16.901104] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:10:16.901111] '
                                            '--match-allocations [2022-10-26 '
                                            '04:10:16.901118] '
                                            '--file-prefix=spdk_pid137279 '
                                            '[2022-10-26 04:10:16.901126] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:10:17.039698] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:10:17.081578] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:10:17.082253] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.output'),
                                  'state': {'begin': 1666743126.8989022,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.096217393875122,
                                            'end': 1666743137.9951196,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_12.state')},
                       'cmd_120': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '510139.78 IOPS     249.09 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '510139.78 IOPS     249.09 MiB/s\n'
                                             '[2022-10-26 04:21:56.963424] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-26 04:21:56.963490] [ '
                                             'DPDK EAL parameters: [2022-10-26 '
                                             '04:21:56.963502] bdevperf '
                                             '[2022-10-26 04:21:56.963511] '
                                             '--no-shconf [2022-10-26 '
                                             '04:21:56.963520] -l 0 '
                                             '[2022-10-26 04:21:56.963527] '
                                             '--huge-unlink [2022-10-26 '
                                             '04:21:56.963536] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-26 04:21:56.963543] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-26 04:21:56.963551] '
                                             '--log-level=user1:6 [2022-10-26 '
                                             '04:21:56.963559] --iova-mode=pa '
                                             '[2022-10-26 04:21:56.963567] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-26 04:21:56.963575] '
                                             '--match-allocations [2022-10-26 '
                                             '04:21:56.963583] '
                                             '--file-prefix=spdk_pid138648 '
                                             '[2022-10-26 04:21:56.963590] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-26 04:21:57.102363] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-26 04:21:57.145569] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-26 04:21:57.146250] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.output'),
                                   'state': {'begin': 1666743826.971447,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '-q 8 -o 512 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.168527841567993,
                                             'end': 1666743838.1399748,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_120.state')},
                       'cmd_13': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.output'),
                                  'state': {'begin': 1666743138.0030165,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7842626571655273,
                                            'end': 1666743138.7872791,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_13.state')},
                       'cmd_14': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '147294.48 IOPS      71.92 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '147294.48 IOPS      71.92 MiB/s\n'
                                            '[2022-10-26 04:10:29.810550] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:10:29.810618] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:10:29.810628] bdevperf '
                                            '[2022-10-26 04:10:29.810637] '
                                            '--no-shconf [2022-10-26 '
                                            '04:10:29.810646] -l 0 [2022-10-26 '
                                            '04:10:29.810653] --huge-unlink '
                                            '[2022-10-26 04:10:29.810662] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:10:29.810670] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:10:29.810677] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:10:29.810685] --iova-mode=pa '
                                            '[2022-10-26 04:10:29.810694] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:10:29.810701] '
                                            '--match-allocations [2022-10-26 '
                                            '04:10:29.810710] '
                                            '--file-prefix=spdk_pid137304 '
                                            '[2022-10-26 04:10:29.810718] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:10:29.949133] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:10:29.993590] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:10:29.994247] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.output'),
                                  'state': {'begin': 1666743139.8103492,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.096240758895874,
                                            'end': 1666743150.90659,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_14.state')},
                       'cmd_15': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.output'),
                                  'state': {'begin': 1666743150.9092145,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7844667434692383,
                                            'end': 1666743151.6936812,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_15.state')},
                       'cmd_16': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '147811.38 IOPS      72.17 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '147811.38 IOPS      72.17 MiB/s\n'
                                            '[2022-10-26 04:10:42.746033] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:10:42.746103] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:10:42.746113] bdevperf '
                                            '[2022-10-26 04:10:42.746122] '
                                            '--no-shconf [2022-10-26 '
                                            '04:10:42.746130] -l 0 [2022-10-26 '
                                            '04:10:42.746139] --huge-unlink '
                                            '[2022-10-26 04:10:42.746146] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:10:42.746154] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:10:42.746161] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:10:42.746169] --iova-mode=pa '
                                            '[2022-10-26 04:10:42.746176] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:10:42.746183] '
                                            '--match-allocations [2022-10-26 '
                                            '04:10:42.746191] '
                                            '--file-prefix=spdk_pid137329 '
                                            '[2022-10-26 04:10:42.746198] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:10:42.884755] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:10:42.934563] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:10:42.935300] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.output'),
                                  'state': {'begin': 1666743152.73093,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.12259316444397,
                                            'end': 1666743163.8535233,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_16.state')},
                       'cmd_17': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.output'),
                                  'state': {'begin': 1666743163.8573275,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7823679447174072,
                                            'end': 1666743164.6396954,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_17.state')},
                       'cmd_18': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '147836.09 IOPS      72.19 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '147836.09 IOPS      72.19 MiB/s\n'
                                            '[2022-10-26 04:10:55.660577] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:10:55.660666] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:10:55.660678] bdevperf '
                                            '[2022-10-26 04:10:55.660688] '
                                            '--no-shconf [2022-10-26 '
                                            '04:10:55.660697] -l 0 [2022-10-26 '
                                            '04:10:55.660706] --huge-unlink '
                                            '[2022-10-26 04:10:55.660716] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:10:55.660724] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:10:55.660733] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:10:55.660742] --iova-mode=pa '
                                            '[2022-10-26 04:10:55.660750] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:10:55.660759] '
                                            '--match-allocations [2022-10-26 '
                                            '04:10:55.660768] '
                                            '--file-prefix=spdk_pid137354 '
                                            '[2022-10-26 04:10:55.660776] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:10:55.802085] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:10:55.849530] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:10:55.850223] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.output'),
                                  'state': {'begin': 1666743165.662862,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.099805116653442,
                                            'end': 1666743176.7626672,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_18.state')},
                       'cmd_19': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.output'),
                                  'state': {'begin': 1666743176.7660782,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7962827682495117,
                                            'end': 1666743177.562361,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_19.state')},
                       'cmd_20': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '145829.56 IOPS      71.21 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '145829.56 IOPS      71.21 MiB/s\n'
                                            '[2022-10-26 04:11:08.598776] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:11:08.598845] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:11:08.598856] bdevperf '
                                            '[2022-10-26 04:11:08.598865] '
                                            '--no-shconf [2022-10-26 '
                                            '04:11:08.598874] -l 0 [2022-10-26 '
                                            '04:11:08.598882] --huge-unlink '
                                            '[2022-10-26 04:11:08.598891] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:11:08.598898] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:11:08.598905] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:11:08.598915] --iova-mode=pa '
                                            '[2022-10-26 04:11:08.598923] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:11:08.598931] '
                                            '--match-allocations [2022-10-26 '
                                            '04:11:08.598939] '
                                            '--file-prefix=spdk_pid137380 '
                                            '[2022-10-26 04:11:08.598947] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:11:08.737667] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:11:08.785519] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:11:08.786203] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.output'),
                                  'state': {'begin': 1666743178.5923176,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.168691635131836,
                                            'end': 1666743189.7610092,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_20.state')},
                       'cmd_21': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.output'),
                                  'state': {'begin': 1666743189.7641158,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7852909564971924,
                                            'end': 1666743190.5494068,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_21.state')},
                       'cmd_22': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '157191.61 IOPS      76.75 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '157191.61 IOPS      76.75 MiB/s\n'
                                            '[2022-10-26 04:11:21.593539] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:11:21.593610] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:11:21.593621] bdevperf '
                                            '[2022-10-26 04:11:21.593629] '
                                            '--no-shconf [2022-10-26 '
                                            '04:11:21.593638] -l 0 [2022-10-26 '
                                            '04:11:21.593646] --huge-unlink '
                                            '[2022-10-26 04:11:21.593653] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:11:21.593661] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:11:21.593669] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:11:21.593677] --iova-mode=pa '
                                            '[2022-10-26 04:11:21.593685] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:11:21.593693] '
                                            '--match-allocations [2022-10-26 '
                                            '04:11:21.593701] '
                                            '--file-prefix=spdk_pid137406 '
                                            '[2022-10-26 04:11:21.593709] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:11:21.732218] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:11:21.777567] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:11:21.778256] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.output'),
                                  'state': {'begin': 1666743191.5794482,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.181533575057983,
                                            'end': 1666743202.7609818,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_22.state')},
                       'cmd_23': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.output'),
                                  'state': {'begin': 1666743202.7638466,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.796776533126831,
                                            'end': 1666743203.5606232,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_23.state')},
                       'cmd_24': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '145943.30 IOPS      71.26 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '145943.30 IOPS      71.26 MiB/s\n'
                                            '[2022-10-26 04:11:34.584729] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:11:34.584800] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:11:34.584811] bdevperf '
                                            '[2022-10-26 04:11:34.584820] '
                                            '--no-shconf [2022-10-26 '
                                            '04:11:34.584829] -l 0 [2022-10-26 '
                                            '04:11:34.584837] --huge-unlink '
                                            '[2022-10-26 04:11:34.584846] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:11:34.584854] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:11:34.584861] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:11:34.584869] --iova-mode=pa '
                                            '[2022-10-26 04:11:34.584877] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:11:34.584885] '
                                            '--match-allocations [2022-10-26 '
                                            '04:11:34.584893] '
                                            '--file-prefix=spdk_pid137431 '
                                            '[2022-10-26 04:11:34.584902] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:11:34.723185] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:11:34.765492] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:11:34.766168] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.output'),
                                  'state': {'begin': 1666743204.5806067,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.164729356765747,
                                            'end': 1666743215.745336,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_24.state')},
                       'cmd_25': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.output'),
                                  'state': {'begin': 1666743215.748288,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.791050910949707,
                                            'end': 1666743216.5393388,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_25.state')},
                       'cmd_26': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '151211.44 IOPS      73.83 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '151211.44 IOPS      73.83 MiB/s\n'
                                            '[2022-10-26 04:11:47.573241] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:11:47.573309] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:11:47.573320] bdevperf '
                                            '[2022-10-26 04:11:47.573329] '
                                            '--no-shconf [2022-10-26 '
                                            '04:11:47.573338] -l 0 [2022-10-26 '
                                            '04:11:47.573346] --huge-unlink '
                                            '[2022-10-26 04:11:47.573355] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:11:47.573363] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:11:47.573389] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:11:47.573398] --iova-mode=pa '
                                            '[2022-10-26 04:11:47.573406] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:11:47.573414] '
                                            '--match-allocations [2022-10-26 '
                                            '04:11:47.573426] '
                                            '--file-prefix=spdk_pid137456 '
                                            '[2022-10-26 04:11:47.573434] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:11:47.711808] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:11:47.753532] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:11:47.754194] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.output'),
                                  'state': {'begin': 1666743217.5643964,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.17328691482544,
                                            'end': 1666743228.7376833,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_26.state')},
                       'cmd_27': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.output'),
                                  'state': {'begin': 1666743228.7405007,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8015680313110352,
                                            'end': 1666743229.5420687,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_27.state')},
                       'cmd_28': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '139449.94 IOPS      68.09 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '139449.94 IOPS      68.09 MiB/s\n'
                                            '[2022-10-26 04:12:00.586747] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:12:00.586818] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:12:00.586829] bdevperf '
                                            '[2022-10-26 04:12:00.586837] '
                                            '--no-shconf [2022-10-26 '
                                            '04:12:00.586845] -l 0 [2022-10-26 '
                                            '04:12:00.586853] --huge-unlink '
                                            '[2022-10-26 04:12:00.586860] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:12:00.586868] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:12:00.586875] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:12:00.586883] --iova-mode=pa '
                                            '[2022-10-26 04:12:00.586890] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:12:00.586898] '
                                            '--match-allocations [2022-10-26 '
                                            '04:12:00.586905] '
                                            '--file-prefix=spdk_pid137481 '
                                            '[2022-10-26 04:12:00.586912] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:12:00.725651] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:12:00.765524] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:12:00.766197] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.output'),
                                  'state': {'begin': 1666743230.5813017,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.189642906188965,
                                            'end': 1666743241.7709446,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_28.state')},
                       'cmd_29': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.output'),
                                  'state': {'begin': 1666743241.7739084,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7814905643463135,
                                            'end': 1666743242.555399,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_29.state')},
                       'cmd_30': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '139549.74 IOPS      68.14 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '139549.74 IOPS      68.14 MiB/s\n'
                                            '[2022-10-26 04:12:13.587573] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:12:13.587648] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:12:13.587659] bdevperf '
                                            '[2022-10-26 04:12:13.587669] '
                                            '--no-shconf [2022-10-26 '
                                            '04:12:13.587678] -l 0 [2022-10-26 '
                                            '04:12:13.587686] --huge-unlink '
                                            '[2022-10-26 04:12:13.587695] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:12:13.587703] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:12:13.587711] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:12:13.587720] --iova-mode=pa '
                                            '[2022-10-26 04:12:13.587728] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:12:13.587736] '
                                            '--match-allocations [2022-10-26 '
                                            '04:12:13.587744] '
                                            '--file-prefix=spdk_pid137506 '
                                            '[2022-10-26 04:12:13.587752] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:12:13.726489] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:12:13.773484] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:12:13.774158] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.output'),
                                  'state': {'begin': 1666743243.586331,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 1 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.183897733688354,
                                            'end': 1666743254.7702286,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_30.state')},
                       'cmd_31': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.output'),
                                  'state': {'begin': 1666743254.7777295,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7828829288482666,
                                            'end': 1666743255.5606124,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_31.state')},
                       'cmd_32': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '262928.44 IOPS     128.38 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '262928.44 IOPS     128.38 MiB/s\n'
                                            '[2022-10-26 04:12:26.587184] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:12:26.587258] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:12:26.587270] bdevperf '
                                            '[2022-10-26 04:12:26.587278] '
                                            '--no-shconf [2022-10-26 '
                                            '04:12:26.587287] -l 0 [2022-10-26 '
                                            '04:12:26.587295] --huge-unlink '
                                            '[2022-10-26 04:12:26.587304] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:12:26.587312] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:12:26.587319] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:12:26.587328] --iova-mode=pa '
                                            '[2022-10-26 04:12:26.587337] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:12:26.587344] '
                                            '--match-allocations [2022-10-26 '
                                            '04:12:26.587352] '
                                            '--file-prefix=spdk_pid137531 '
                                            '[2022-10-26 04:12:26.587360] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:12:26.725751] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:12:26.765496] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:12:26.766167] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.output'),
                                  'state': {'begin': 1666743256.5828512,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.110819339752197,
                                            'end': 1666743267.6936705,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_32.state')},
                       'cmd_33': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.output'),
                                  'state': {'begin': 1666743267.696667,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8754153251647949,
                                            'end': 1666743268.5720823,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_33.state')},
                       'cmd_34': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '242424.58 IOPS     118.37 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '242424.58 IOPS     118.37 MiB/s\n'
                                            '[2022-10-26 04:12:39.856534] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:12:39.856602] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:12:39.856613] bdevperf '
                                            '[2022-10-26 04:12:39.856621] '
                                            '--no-shconf [2022-10-26 '
                                            '04:12:39.856629] -l 0 [2022-10-26 '
                                            '04:12:39.856637] --huge-unlink '
                                            '[2022-10-26 04:12:39.856644] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:12:39.856652] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:12:39.856659] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:12:39.856667] --iova-mode=pa '
                                            '[2022-10-26 04:12:39.856674] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:12:39.856681] '
                                            '--match-allocations [2022-10-26 '
                                            '04:12:39.856688] '
                                            '--file-prefix=spdk_pid137556 '
                                            '[2022-10-26 04:12:39.856696] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:12:39.994898] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:12:40.041502] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:12:40.042162] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.output'),
                                  'state': {'begin': 1666743269.7216797,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.253293752670288,
                                            'end': 1666743280.9749734,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_34.state')},
                       'cmd_35': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.output'),
                                  'state': {'begin': 1666743280.9778373,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7800905704498291,
                                            'end': 1666743281.757928,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_35.state')},
                       'cmd_36': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '263549.12 IOPS     128.69 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '263549.12 IOPS     128.69 MiB/s\n'
                                            '[2022-10-26 04:12:52.784402] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:12:52.784475] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:12:52.784486] bdevperf '
                                            '[2022-10-26 04:12:52.784494] '
                                            '--no-shconf [2022-10-26 '
                                            '04:12:52.784503] -l 0 [2022-10-26 '
                                            '04:12:52.784511] --huge-unlink '
                                            '[2022-10-26 04:12:52.784520] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:12:52.784527] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:12:52.784536] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:12:52.784544] --iova-mode=pa '
                                            '[2022-10-26 04:12:52.784552] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:12:52.784560] '
                                            '--match-allocations [2022-10-26 '
                                            '04:12:52.784568] '
                                            '--file-prefix=spdk_pid137581 '
                                            '[2022-10-26 04:12:52.784576] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:12:52.923433] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:12:52.969615] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:12:52.970299] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.output'),
                                  'state': {'begin': 1666743282.7862253,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.112562894821167,
                                            'end': 1666743293.8987882,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_36.state')},
                       'cmd_37': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.output'),
                                  'state': {'begin': 1666743293.9017942,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7872776985168457,
                                            'end': 1666743294.689072,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_37.state')},
                       'cmd_38': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '228150.59 IOPS     111.40 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '228150.59 IOPS     111.40 MiB/s\n'
                                            '[2022-10-26 04:13:05.723004] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:13:05.723072] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:13:05.723082] bdevperf '
                                            '[2022-10-26 04:13:05.723091] '
                                            '--no-shconf [2022-10-26 '
                                            '04:13:05.723100] -l 0 [2022-10-26 '
                                            '04:13:05.723108] --huge-unlink '
                                            '[2022-10-26 04:13:05.723127] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:13:05.723136] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:13:05.723143] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:13:05.723151] --iova-mode=pa '
                                            '[2022-10-26 04:13:05.723159] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:13:05.723167] '
                                            '--match-allocations [2022-10-26 '
                                            '04:13:05.723176] '
                                            '--file-prefix=spdk_pid137607 '
                                            '[2022-10-26 04:13:05.723184] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:13:05.862734] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:13:05.917790] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:13:05.918471] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.output'),
                                  'state': {'begin': 1666743295.7155757,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.114653825759888,
                                            'end': 1666743306.8302295,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_38.state')},
                       'cmd_39': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.output'),
                                  'state': {'begin': 1666743306.8331847,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.784264087677002,
                                            'end': 1666743307.6174488,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_39.state')},
                       'cmd_40': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '230398.16 IOPS     112.50 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '230398.16 IOPS     112.50 MiB/s\n'
                                            '[2022-10-26 04:13:18.651302] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:13:18.651371] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:13:18.651381] bdevperf '
                                            '[2022-10-26 04:13:18.651390] '
                                            '--no-shconf [2022-10-26 '
                                            '04:13:18.651399] -l 0 [2022-10-26 '
                                            '04:13:18.651407] --huge-unlink '
                                            '[2022-10-26 04:13:18.651416] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:13:18.651423] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:13:18.651431] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:13:18.651440] --iova-mode=pa '
                                            '[2022-10-26 04:13:18.651448] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:13:18.651456] '
                                            '--match-allocations [2022-10-26 '
                                            '04:13:18.651464] '
                                            '--file-prefix=spdk_pid137632 '
                                            '[2022-10-26 04:13:18.651472] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:13:18.790232] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:13:18.829548] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:13:18.830238] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.output'),
                                  'state': {'begin': 1666743308.6497045,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.093471765518188,
                                            'end': 1666743319.7431762,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_40.state')},
                       'cmd_41': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.output'),
                                  'state': {'begin': 1666743319.7462144,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7990851402282715,
                                            'end': 1666743320.5452995,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_41.state')},
                       'cmd_42': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '229316.75 IOPS     111.97 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '229316.75 IOPS     111.97 MiB/s\n'
                                            '[2022-10-26 04:13:31.605298] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:13:31.605366] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:13:31.605377] bdevperf '
                                            '[2022-10-26 04:13:31.605386] '
                                            '--no-shconf [2022-10-26 '
                                            '04:13:31.605394] -l 0 [2022-10-26 '
                                            '04:13:31.605402] --huge-unlink '
                                            '[2022-10-26 04:13:31.605411] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:13:31.605419] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:13:31.605426] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:13:31.605434] --iova-mode=pa '
                                            '[2022-10-26 04:13:31.605441] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:13:31.605449] '
                                            '--match-allocations [2022-10-26 '
                                            '04:13:31.605456] '
                                            '--file-prefix=spdk_pid137657 '
                                            '[2022-10-26 04:13:31.605464] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:13:31.743848] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:13:31.793535] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:13:31.794215] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.output'),
                                  'state': {'begin': 1666743321.5992064,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.111653804779053,
                                            'end': 1666743332.7108603,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_42.state')},
                       'cmd_43': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.output'),
                                  'state': {'begin': 1666743332.713505,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7880523204803467,
                                            'end': 1666743333.5015574,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_43.state')},
                       'cmd_44': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '291240.87 IOPS     142.21 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '291240.87 IOPS     142.21 MiB/s\n'
                                            '[2022-10-26 04:13:44.534674] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:13:44.534744] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:13:44.534755] bdevperf '
                                            '[2022-10-26 04:13:44.534764] '
                                            '--no-shconf [2022-10-26 '
                                            '04:13:44.534773] -l 0 [2022-10-26 '
                                            '04:13:44.534780] --huge-unlink '
                                            '[2022-10-26 04:13:44.534789] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:13:44.534797] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:13:44.534805] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:13:44.534819] --iova-mode=pa '
                                            '[2022-10-26 04:13:44.534827] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:13:44.534835] '
                                            '--match-allocations [2022-10-26 '
                                            '04:13:44.534842] '
                                            '--file-prefix=spdk_pid137682 '
                                            '[2022-10-26 04:13:44.534850] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:13:44.673093] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:13:44.725564] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:13:44.726241] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.output'),
                                  'state': {'begin': 1666743334.5314794,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.119513273239136,
                                            'end': 1666743345.6509926,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_44.state')},
                       'cmd_45': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.output'),
                                  'state': {'begin': 1666743345.6540956,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7841320037841797,
                                            'end': 1666743346.4382277,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_45.state')},
                       'cmd_46': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '290441.54 IOPS     141.82 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '290441.54 IOPS     141.82 MiB/s\n'
                                            '[2022-10-26 04:13:57.478245] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:13:57.478314] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:13:57.478324] bdevperf '
                                            '[2022-10-26 04:13:57.478333] '
                                            '--no-shconf [2022-10-26 '
                                            '04:13:57.478341] -l 0 [2022-10-26 '
                                            '04:13:57.478349] --huge-unlink '
                                            '[2022-10-26 04:13:57.478357] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:13:57.478365] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:13:57.478372] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:13:57.478380] --iova-mode=pa '
                                            '[2022-10-26 04:13:57.478387] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:13:57.478395] '
                                            '--match-allocations [2022-10-26 '
                                            '04:13:57.478402] '
                                            '--file-prefix=spdk_pid137708 '
                                            '[2022-10-26 04:13:57.478409] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:13:57.617477] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:13:57.661885] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:13:57.662595] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.output'),
                                  'state': {'begin': 1666743347.4800632,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.106460094451904,
                                            'end': 1666743358.5865233,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_46.state')},
                       'cmd_47': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.output'),
                                  'state': {'begin': 1666743358.5895798,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.786015510559082,
                                            'end': 1666743359.3755953,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_47.state')},
                       'cmd_48': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '290486.27 IOPS     141.84 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '290486.27 IOPS     141.84 MiB/s\n'
                                            '[2022-10-26 04:14:10.393432] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:14:10.393501] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:14:10.393511] bdevperf '
                                            '[2022-10-26 04:14:10.393520] '
                                            '--no-shconf [2022-10-26 '
                                            '04:14:10.393528] -l 0 [2022-10-26 '
                                            '04:14:10.393536] --huge-unlink '
                                            '[2022-10-26 04:14:10.393544] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:14:10.393552] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:14:10.393559] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:14:10.393567] --iova-mode=pa '
                                            '[2022-10-26 04:14:10.393575] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:14:10.393583] '
                                            '--match-allocations [2022-10-26 '
                                            '04:14:10.393590] '
                                            '--file-prefix=spdk_pid137735 '
                                            '[2022-10-26 04:14:10.393598] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:14:10.531814] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:14:10.581554] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:14:10.582236] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.output'),
                                  'state': {'begin': 1666743360.389052,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.11704397201538,
                                            'end': 1666743371.506096,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_48.state')},
                       'cmd_49': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.output'),
                                  'state': {'begin': 1666743371.509128,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7821986675262451,
                                            'end': 1666743372.2913268,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_49.state')},
                       'cmd_50': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '285675.37 IOPS     139.49 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '285675.37 IOPS     139.49 MiB/s\n'
                                            '[2022-10-26 04:14:23.317220] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:14:23.317292] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:14:23.317303] bdevperf '
                                            '[2022-10-26 04:14:23.317312] '
                                            '--no-shconf [2022-10-26 '
                                            '04:14:23.317320] -l 0 [2022-10-26 '
                                            '04:14:23.317329] --huge-unlink '
                                            '[2022-10-26 04:14:23.317338] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:14:23.317345] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:14:23.317352] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:14:23.317360] --iova-mode=pa '
                                            '[2022-10-26 04:14:23.317367] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:14:23.317375] '
                                            '--match-allocations [2022-10-26 '
                                            '04:14:23.317383] '
                                            '--file-prefix=spdk_pid137760 '
                                            '[2022-10-26 04:14:23.317391] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:14:23.456191] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:14:23.509543] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:14:23.510240] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.output'),
                                  'state': {'begin': 1666743373.3197203,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.168123722076416,
                                            'end': 1666743384.487844,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_50.state')},
                       'cmd_51': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.output'),
                                  'state': {'begin': 1666743384.4903607,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7734940052032471,
                                            'end': 1666743385.2638547,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_51.state')},
                       'cmd_52': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '285787.73 IOPS     139.54 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '285787.73 IOPS     139.54 MiB/s\n'
                                            '[2022-10-26 04:14:36.284758] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:14:36.284827] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:14:36.284838] bdevperf '
                                            '[2022-10-26 04:14:36.284847] '
                                            '--no-shconf [2022-10-26 '
                                            '04:14:36.284855] -l 0 [2022-10-26 '
                                            '04:14:36.284864] --huge-unlink '
                                            '[2022-10-26 04:14:36.284873] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:14:36.284880] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:14:36.284888] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:14:36.284896] --iova-mode=pa '
                                            '[2022-10-26 04:14:36.284905] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:14:36.284913] '
                                            '--match-allocations [2022-10-26 '
                                            '04:14:36.284921] '
                                            '--file-prefix=spdk_pid137786 '
                                            '[2022-10-26 04:14:36.284929] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:14:36.423316] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:14:36.465562] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:14:36.466223] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.output'),
                                  'state': {'begin': 1666743386.2882006,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.159849882125854,
                                            'end': 1666743397.4480505,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_52.state')},
                       'cmd_53': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.output'),
                                  'state': {'begin': 1666743397.4523597,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7845513820648193,
                                            'end': 1666743398.236911,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_53.state')},
                       'cmd_54': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '283717.83 IOPS     138.53 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '283717.83 IOPS     138.53 MiB/s\n'
                                            '[2022-10-26 04:14:49.262571] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:14:49.262640] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:14:49.262651] bdevperf '
                                            '[2022-10-26 04:14:49.262661] '
                                            '--no-shconf [2022-10-26 '
                                            '04:14:49.262669] -l 0 [2022-10-26 '
                                            '04:14:49.262676] --huge-unlink '
                                            '[2022-10-26 04:14:49.262684] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:14:49.262692] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:14:49.262699] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:14:49.262706] --iova-mode=pa '
                                            '[2022-10-26 04:14:49.262714] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:14:49.262722] '
                                            '--match-allocations [2022-10-26 '
                                            '04:14:49.262729] '
                                            '--file-prefix=spdk_pid137811 '
                                            '[2022-10-26 04:14:49.262737] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:14:49.401169] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:14:49.449515] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:14:49.450200] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.output'),
                                  'state': {'begin': 1666743399.2636812,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.170012474060059,
                                            'end': 1666743410.4336936,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_54.state')},
                       'cmd_55': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.output'),
                                  'state': {'begin': 1666743410.4362636,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7805120944976807,
                                            'end': 1666743411.2167757,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_55.state')},
                       'cmd_56': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '273786.89 IOPS     133.69 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '273786.89 IOPS     133.69 MiB/s\n'
                                            '[2022-10-26 04:15:02.256385] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:15:02.256453] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:15:02.256464] bdevperf '
                                            '[2022-10-26 04:15:02.256473] '
                                            '--no-shconf [2022-10-26 '
                                            '04:15:02.256481] -l 0 [2022-10-26 '
                                            '04:15:02.256489] --huge-unlink '
                                            '[2022-10-26 04:15:02.256498] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:15:02.256506] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:15:02.256514] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:15:02.256522] --iova-mode=pa '
                                            '[2022-10-26 04:15:02.256531] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:15:02.256539] '
                                            '--match-allocations [2022-10-26 '
                                            '04:15:02.256548] '
                                            '--file-prefix=spdk_pid137838 '
                                            '[2022-10-26 04:15:02.256556] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:15:02.395084] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:15:02.453546] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:15:02.454235] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.output'),
                                  'state': {'begin': 1666743412.2478497,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.20577597618103,
                                            'end': 1666743423.4536257,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_56.state')},
                       'cmd_57': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.output'),
                                  'state': {'begin': 1666743423.456269,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7816627025604248,
                                            'end': 1666743424.2379317,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_57.state')},
                       'cmd_58': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '273253.44 IOPS     133.42 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '273253.44 IOPS     133.42 MiB/s\n'
                                            '[2022-10-26 04:15:15.273977] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:15:15.274047] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:15:15.274058] bdevperf '
                                            '[2022-10-26 04:15:15.274067] '
                                            '--no-shconf [2022-10-26 '
                                            '04:15:15.274075] -l 0 [2022-10-26 '
                                            '04:15:15.274082] --huge-unlink '
                                            '[2022-10-26 04:15:15.274091] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:15:15.274098] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:15:15.274105] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:15:15.274120] --iova-mode=pa '
                                            '[2022-10-26 04:15:15.274128] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:15:15.274135] '
                                            '--match-allocations [2022-10-26 '
                                            '04:15:15.274142] '
                                            '--file-prefix=spdk_pid137864 '
                                            '[2022-10-26 04:15:15.274150] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:15:15.412303] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:15:15.453532] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:15:15.454217] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.output'),
                                  'state': {'begin': 1666743425.2605772,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.19258713722229,
                                            'end': 1666743436.4531643,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_58.state')},
                       'cmd_59': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.output'),
                                  'state': {'begin': 1666743436.4555464,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7853162288665771,
                                            'end': 1666743437.2408626,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_59.state')},
                       'cmd_60': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '273765.54 IOPS     133.67 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '273765.54 IOPS     133.67 MiB/s\n'
                                            '[2022-10-26 04:15:28.271983] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:15:28.272054] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:15:28.272065] bdevperf '
                                            '[2022-10-26 04:15:28.272073] '
                                            '--no-shconf [2022-10-26 '
                                            '04:15:28.272081] -l 0 [2022-10-26 '
                                            '04:15:28.272089] --huge-unlink '
                                            '[2022-10-26 04:15:28.272098] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:15:28.272106] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:15:28.272114] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:15:28.272122] --iova-mode=pa '
                                            '[2022-10-26 04:15:28.272130] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:15:28.272137] '
                                            '--match-allocations [2022-10-26 '
                                            '04:15:28.272145] '
                                            '--file-prefix=spdk_pid137889 '
                                            '[2022-10-26 04:15:28.272153] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:15:28.410856] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:15:28.457568] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:15:28.458231] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.output'),
                                  'state': {'begin': 1666743438.2684667,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 2 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.178375244140625,
                                            'end': 1666743449.446842,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_60.state')},
                       'cmd_61': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.output'),
                                  'state': {'begin': 1666743449.4493637,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7808904647827148,
                                            'end': 1666743450.2302542,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_61.state')},
                       'cmd_62': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '392406.39 IOPS     191.60 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '392406.39 IOPS     191.60 MiB/s\n'
                                            '[2022-10-26 04:15:41.256135] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:15:41.256203] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:15:41.256213] bdevperf '
                                            '[2022-10-26 04:15:41.256222] '
                                            '--no-shconf [2022-10-26 '
                                            '04:15:41.256230] -l 0 [2022-10-26 '
                                            '04:15:41.256238] --huge-unlink '
                                            '[2022-10-26 04:15:41.256246] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:15:41.256253] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:15:41.256260] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:15:41.256268] --iova-mode=pa '
                                            '[2022-10-26 04:15:41.256276] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:15:41.256283] '
                                            '--match-allocations [2022-10-26 '
                                            '04:15:41.256290] '
                                            '--file-prefix=spdk_pid137914 '
                                            '[2022-10-26 04:15:41.256298] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:15:41.394453] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:15:41.441506] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:15:41.442191] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.output'),
                                  'state': {'begin': 1666743451.2566087,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.140043497085571,
                                            'end': 1666743462.3966522,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_62.state')},
                       'cmd_63': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.output'),
                                  'state': {'begin': 1666743462.4036922,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7801389694213867,
                                            'end': 1666743463.1838312,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_63.state')},
                       'cmd_64': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '392406.89 IOPS     191.60 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '392406.89 IOPS     191.60 MiB/s\n'
                                            '[2022-10-26 04:15:54.215274] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:15:54.215345] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:15:54.215356] bdevperf '
                                            '[2022-10-26 04:15:54.215365] '
                                            '--no-shconf [2022-10-26 '
                                            '04:15:54.215374] -l 0 [2022-10-26 '
                                            '04:15:54.215382] --huge-unlink '
                                            '[2022-10-26 04:15:54.215390] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:15:54.215398] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:15:54.215406] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:15:54.215414] --iova-mode=pa '
                                            '[2022-10-26 04:15:54.215423] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:15:54.215431] '
                                            '--match-allocations [2022-10-26 '
                                            '04:15:54.215439] '
                                            '--file-prefix=spdk_pid137939 '
                                            '[2022-10-26 04:15:54.215448] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:15:54.353784] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:15:54.405744] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:15:54.406405] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.output'),
                                  'state': {'begin': 1666743464.210217,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.134780406951904,
                                            'end': 1666743475.3449974,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_64.state')},
                       'cmd_65': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.output'),
                                  'state': {'begin': 1666743475.3473835,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.787315845489502,
                                            'end': 1666743476.1346993,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_65.state')},
                       'cmd_66': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '392071.55 IOPS     191.44 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '392071.55 IOPS     191.44 MiB/s\n'
                                            '[2022-10-26 04:16:07.163390] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:16:07.163460] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:16:07.163471] bdevperf '
                                            '[2022-10-26 04:16:07.163480] '
                                            '--no-shconf [2022-10-26 '
                                            '04:16:07.163488] -l 0 [2022-10-26 '
                                            '04:16:07.163496] --huge-unlink '
                                            '[2022-10-26 04:16:07.163503] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:16:07.163511] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:16:07.163518] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:16:07.163526] --iova-mode=pa '
                                            '[2022-10-26 04:16:07.163533] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:16:07.163541] '
                                            '--match-allocations [2022-10-26 '
                                            '04:16:07.163548] '
                                            '--file-prefix=spdk_pid137964 '
                                            '[2022-10-26 04:16:07.163555] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:16:07.302310] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:16:07.345541] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:16:07.346210] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.output'),
                                  'state': {'begin': 1666743477.1620119,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.115491151809692,
                                            'end': 1666743488.277503,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_66.state')},
                       'cmd_67': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.output'),
                                  'state': {'begin': 1666743488.2798777,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7826259136199951,
                                            'end': 1666743489.0625036,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_67.state')},
                       'cmd_68': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '430379.28 IOPS     210.15 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '430379.28 IOPS     210.15 MiB/s\n'
                                            '[2022-10-26 04:16:20.087240] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:16:20.087312] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:16:20.087323] bdevperf '
                                            '[2022-10-26 04:16:20.087332] '
                                            '--no-shconf [2022-10-26 '
                                            '04:16:20.087340] -l 0 [2022-10-26 '
                                            '04:16:20.087348] --huge-unlink '
                                            '[2022-10-26 04:16:20.087357] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:16:20.087364] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:16:20.087372] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:16:20.087380] --iova-mode=pa '
                                            '[2022-10-26 04:16:20.087388] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:16:20.087396] '
                                            '--match-allocations [2022-10-26 '
                                            '04:16:20.087404] '
                                            '--file-prefix=spdk_pid137989 '
                                            '[2022-10-26 04:16:20.087413] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:16:20.225818] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:16:20.269590] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:16:20.270271] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.output'),
                                  'state': {'begin': 1666743490.08891,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.103523015975952,
                                            'end': 1666743501.192433,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_68.state')},
                       'cmd_69': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.output'),
                                  'state': {'begin': 1666743501.1953654,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.784801721572876,
                                            'end': 1666743501.9801672,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_69.state')},
                       'cmd_70': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '431736.81 IOPS     210.81 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '431736.81 IOPS     210.81 MiB/s\n'
                                            '[2022-10-26 04:16:33.002825] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:16:33.002895] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:16:33.002905] bdevperf '
                                            '[2022-10-26 04:16:33.002915] '
                                            '--no-shconf [2022-10-26 '
                                            '04:16:33.002923] -l 0 [2022-10-26 '
                                            '04:16:33.002930] --huge-unlink '
                                            '[2022-10-26 04:16:33.002938] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:16:33.002946] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:16:33.002954] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:16:33.002962] --iova-mode=pa '
                                            '[2022-10-26 04:16:33.002970] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:16:33.002977] '
                                            '--match-allocations [2022-10-26 '
                                            '04:16:33.002985] '
                                            '--file-prefix=spdk_pid138014 '
                                            '[2022-10-26 04:16:33.002992] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:16:33.141476] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:16:33.189533] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:16:33.190204] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.output'),
                                  'state': {'begin': 1666743503.006869,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.098368167877197,
                                            'end': 1666743514.1052372,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_70.state')},
                       'cmd_71': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.output'),
                                  'state': {'begin': 1666743514.1128907,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7827651500701904,
                                            'end': 1666743514.8956559,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_71.state')},
                       'cmd_72': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '428711.29 IOPS     209.33 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '428711.29 IOPS     209.33 MiB/s\n'
                                            '[2022-10-26 04:16:45.923337] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:16:45.923406] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:16:45.923416] bdevperf '
                                            '[2022-10-26 04:16:45.923425] '
                                            '--no-shconf [2022-10-26 '
                                            '04:16:45.923433] -l 0 [2022-10-26 '
                                            '04:16:45.923441] --huge-unlink '
                                            '[2022-10-26 04:16:45.923449] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:16:45.923456] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:16:45.923464] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:16:45.923471] --iova-mode=pa '
                                            '[2022-10-26 04:16:45.923479] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:16:45.923486] '
                                            '--match-allocations [2022-10-26 '
                                            '04:16:45.923494] '
                                            '--file-prefix=spdk_pid138039 '
                                            '[2022-10-26 04:16:45.923501] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:16:46.062040] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:16:46.101512] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:16:46.102194] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.output'),
                                  'state': {'begin': 1666743515.9263752,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.100262880325317,
                                            'end': 1666743527.026638,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_72.state')},
                       'cmd_73': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.output'),
                                  'state': {'begin': 1666743527.0294874,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7903826236724854,
                                            'end': 1666743527.81987,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_73.state')},
                       'cmd_74': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '485865.81 IOPS     237.24 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '485865.81 IOPS     237.24 MiB/s\n'
                                            '[2022-10-26 04:16:58.841718] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:16:58.841789] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:16:58.841800] bdevperf '
                                            '[2022-10-26 04:16:58.841809] '
                                            '--no-shconf [2022-10-26 '
                                            '04:16:58.841817] -l 0 [2022-10-26 '
                                            '04:16:58.841825] --huge-unlink '
                                            '[2022-10-26 04:16:58.841833] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:16:58.841840] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:16:58.841848] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:16:58.841855] --iova-mode=pa '
                                            '[2022-10-26 04:16:58.841863] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:16:58.841870] '
                                            '--match-allocations [2022-10-26 '
                                            '04:16:58.841878] '
                                            '--file-prefix=spdk_pid138064 '
                                            '[2022-10-26 04:16:58.841885] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:16:58.980698] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:16:59.025507] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:16:59.026173] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.output'),
                                  'state': {'begin': 1666743528.8436036,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.105374336242676,
                                            'end': 1666743539.948978,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_74.state')},
                       'cmd_75': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.output'),
                                  'state': {'begin': 1666743539.9519782,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7835664749145508,
                                            'end': 1666743540.7355447,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_75.state')},
                       'cmd_76': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '482781.41 IOPS     235.73 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '482781.41 IOPS     235.73 MiB/s\n'
                                            '[2022-10-26 04:17:11.766933] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:17:11.767002] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:17:11.767013] bdevperf '
                                            '[2022-10-26 04:17:11.767022] '
                                            '--no-shconf [2022-10-26 '
                                            '04:17:11.767030] -l 0 [2022-10-26 '
                                            '04:17:11.767038] --huge-unlink '
                                            '[2022-10-26 04:17:11.767047] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:17:11.767055] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:17:11.767062] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:17:11.767071] --iova-mode=pa '
                                            '[2022-10-26 04:17:11.767079] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:17:11.767087] '
                                            '--match-allocations [2022-10-26 '
                                            '04:17:11.767095] '
                                            '--file-prefix=spdk_pid138092 '
                                            '[2022-10-26 04:17:11.767103] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:17:11.905315] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:17:11.945540] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:17:11.946218] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.output'),
                                  'state': {'begin': 1666743541.7653217,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.098800420761108,
                                            'end': 1666743552.8641222,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_76.state')},
                       'cmd_77': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.output'),
                                  'state': {'begin': 1666743552.8684063,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7847492694854736,
                                            'end': 1666743553.6531556,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_77.state')},
                       'cmd_78': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '482648.51 IOPS     235.67 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '482648.51 IOPS     235.67 MiB/s\n'
                                            '[2022-10-26 04:17:24.672976] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:17:24.673047] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:17:24.673057] bdevperf '
                                            '[2022-10-26 04:17:24.673066] '
                                            '--no-shconf [2022-10-26 '
                                            '04:17:24.673074] -l 0 [2022-10-26 '
                                            '04:17:24.673082] --huge-unlink '
                                            '[2022-10-26 04:17:24.673089] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:17:24.673097] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:17:24.673104] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:17:24.673112] --iova-mode=pa '
                                            '[2022-10-26 04:17:24.673119] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:17:24.673126] '
                                            '--match-allocations [2022-10-26 '
                                            '04:17:24.673134] '
                                            '--file-prefix=spdk_pid138117 '
                                            '[2022-10-26 04:17:24.673141] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:17:24.812469] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:17:24.861970] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:17:24.862650] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.output'),
                                  'state': {'begin': 1666743554.6805289,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.108688592910767,
                                            'end': 1666743565.7892175,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_78.state')},
                       'cmd_79': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.output'),
                                  'state': {'begin': 1666743565.7918122,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7834320068359375,
                                            'end': 1666743566.5752442,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_79.state')},
                       'cmd_80': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '479021.22 IOPS     233.90 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '479021.22 IOPS     233.90 MiB/s\n'
                                            '[2022-10-26 04:17:37.615480] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:17:37.615550] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:17:37.615561] bdevperf '
                                            '[2022-10-26 04:17:37.615570] '
                                            '--no-shconf [2022-10-26 '
                                            '04:17:37.615578] -l 0 [2022-10-26 '
                                            '04:17:37.615586] --huge-unlink '
                                            '[2022-10-26 04:17:37.615594] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:17:37.615601] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:17:37.615608] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:17:37.615616] --iova-mode=pa '
                                            '[2022-10-26 04:17:37.615623] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:17:37.615631] '
                                            '--match-allocations [2022-10-26 '
                                            '04:17:37.615639] '
                                            '--file-prefix=spdk_pid138142 '
                                            '[2022-10-26 04:17:37.615647] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:17:37.754205] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:17:37.797538] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:17:37.798220] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.output'),
                                  'state': {'begin': 1666743567.6054509,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.18461012840271,
                                            'end': 1666743578.790061,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_80.state')},
                       'cmd_81': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.output'),
                                  'state': {'begin': 1666743578.7931554,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7960965633392334,
                                            'end': 1666743579.589252,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_81.state')},
                       'cmd_82': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '479430.62 IOPS     234.10 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '479430.62 IOPS     234.10 MiB/s\n'
                                            '[2022-10-26 04:17:50.615507] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:17:50.615575] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:17:50.615586] bdevperf '
                                            '[2022-10-26 04:17:50.615594] '
                                            '--no-shconf [2022-10-26 '
                                            '04:17:50.615602] -l 0 [2022-10-26 '
                                            '04:17:50.615610] --huge-unlink '
                                            '[2022-10-26 04:17:50.615617] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:17:50.615625] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:17:50.615632] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:17:50.615640] --iova-mode=pa '
                                            '[2022-10-26 04:17:50.615648] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:17:50.615655] '
                                            '--match-allocations [2022-10-26 '
                                            '04:17:50.615663] '
                                            '--file-prefix=spdk_pid138167 '
                                            '[2022-10-26 04:17:50.615670] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:17:50.752773] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:17:50.805554] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:17:50.806237] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.output'),
                                  'state': {'begin': 1666743580.6223962,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.173489809036255,
                                            'end': 1666743591.795886,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_82.state')},
                       'cmd_83': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.output'),
                                  'state': {'begin': 1666743591.7985127,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7782433032989502,
                                            'end': 1666743592.576756,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_83.state')},
                       'cmd_84': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '480153.62 IOPS     234.45 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '480153.62 IOPS     234.45 MiB/s\n'
                                            '[2022-10-26 04:18:03.607723] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:18:03.607800] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:18:03.607811] bdevperf '
                                            '[2022-10-26 04:18:03.607820] '
                                            '--no-shconf [2022-10-26 '
                                            '04:18:03.607828] -l 0 [2022-10-26 '
                                            '04:18:03.607836] --huge-unlink '
                                            '[2022-10-26 04:18:03.607844] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:18:03.607852] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:18:03.607859] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:18:03.607868] --iova-mode=pa '
                                            '[2022-10-26 04:18:03.607876] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:18:03.607884] '
                                            '--match-allocations [2022-10-26 '
                                            '04:18:03.607892] '
                                            '--file-prefix=spdk_pid138192 '
                                            '[2022-10-26 04:18:03.607900] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:18:03.746385] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:18:03.789546] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:18:03.790227] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.output'),
                                  'state': {'begin': 1666743593.607161,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.168599128723145,
                                            'end': 1666743604.7757602,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_84.state')},
                       'cmd_85': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.output'),
                                  'state': {'begin': 1666743604.7787178,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7873766422271729,
                                            'end': 1666743605.5660944,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_85.state')},
                       'cmd_86': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '430950.51 IOPS     210.43 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '430950.51 IOPS     210.43 MiB/s\n'
                                            '[2022-10-26 04:18:16.597404] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:18:16.597472] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:18:16.597483] bdevperf '
                                            '[2022-10-26 04:18:16.597491] '
                                            '--no-shconf [2022-10-26 '
                                            '04:18:16.597499] -l 0 [2022-10-26 '
                                            '04:18:16.597507] --huge-unlink '
                                            '[2022-10-26 04:18:16.597516] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:18:16.597524] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:18:16.597531] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:18:16.597557] --iova-mode=pa '
                                            '[2022-10-26 04:18:16.597567] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:18:16.597575] '
                                            '--match-allocations [2022-10-26 '
                                            '04:18:16.597583] '
                                            '--file-prefix=spdk_pid138217 '
                                            '[2022-10-26 04:18:16.597599] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:18:16.735774] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:18:16.777548] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:18:16.778223] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.output'),
                                  'state': {'begin': 1666743606.5988963,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.173280954360962,
                                            'end': 1666743617.7721772,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_86.state')},
                       'cmd_87': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.output'),
                                  'state': {'begin': 1666743617.7751496,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7903523445129395,
                                            'end': 1666743618.565502,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_87.state')},
                       'cmd_88': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '429672.51 IOPS     209.80 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '429672.51 IOPS     209.80 MiB/s\n'
                                            '[2022-10-26 04:18:29.614303] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:18:29.614376] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:18:29.614387] bdevperf '
                                            '[2022-10-26 04:18:29.614395] '
                                            '--no-shconf [2022-10-26 '
                                            '04:18:29.614403] -l 0 [2022-10-26 '
                                            '04:18:29.614411] --huge-unlink '
                                            '[2022-10-26 04:18:29.614419] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:18:29.614426] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:18:29.614433] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:18:29.614441] --iova-mode=pa '
                                            '[2022-10-26 04:18:29.614448] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:18:29.614455] '
                                            '--match-allocations [2022-10-26 '
                                            '04:18:29.614463] '
                                            '--file-prefix=spdk_pid138243 '
                                            '[2022-10-26 04:18:29.614470] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:18:29.753633] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:18:29.801578] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:18:29.802241] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.output'),
                                  'state': {'begin': 1666743619.6014123,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.215760231018066,
                                            'end': 1666743630.8171725,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_88.state')},
                       'cmd_89': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.output'),
                                  'state': {'begin': 1666743630.8198419,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7805182933807373,
                                            'end': 1666743631.6003602,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_89.state')},
                       'cmd_90': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '430511.26 IOPS     210.21 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '430511.26 IOPS     210.21 MiB/s\n'
                                            '[2022-10-26 04:18:42.616257] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:18:42.616325] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:18:42.616336] bdevperf '
                                            '[2022-10-26 04:18:42.616345] '
                                            '--no-shconf [2022-10-26 '
                                            '04:18:42.616353] -l 0 [2022-10-26 '
                                            '04:18:42.616362] --huge-unlink '
                                            '[2022-10-26 04:18:42.616370] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:18:42.616378] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:18:42.616386] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:18:42.616394] --iova-mode=pa '
                                            '[2022-10-26 04:18:42.616410] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:18:42.616419] '
                                            '--match-allocations [2022-10-26 '
                                            '04:18:42.616427] '
                                            '--file-prefix=spdk_pid138269 '
                                            '[2022-10-26 04:18:42.616435] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:18:42.754809] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:18:42.801477] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:18:42.802142] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.output'),
                                  'state': {'begin': 1666743632.626331,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 4 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.178959846496582,
                                            'end': 1666743643.805291,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_90.state')},
                       'cmd_91': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.output'),
                                  'state': {'begin': 1666743643.8072298,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7829594612121582,
                                            'end': 1666743644.5901892,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_91.state')},
                       'cmd_92': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '616823.81 IOPS     301.18 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '616823.81 IOPS     301.18 MiB/s\n'
                                            '[2022-10-26 04:18:55.614634] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:18:55.614701] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:18:55.614712] bdevperf '
                                            '[2022-10-26 04:18:55.614722] '
                                            '--no-shconf [2022-10-26 '
                                            '04:18:55.614730] -l 0 [2022-10-26 '
                                            '04:18:55.614737] --huge-unlink '
                                            '[2022-10-26 04:18:55.614745] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:18:55.614752] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:18:55.614759] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:18:55.614767] --iova-mode=pa '
                                            '[2022-10-26 04:18:55.614774] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:18:55.614782] '
                                            '--match-allocations [2022-10-26 '
                                            '04:18:55.614789] '
                                            '--file-prefix=spdk_pid138294 '
                                            '[2022-10-26 04:18:55.614796] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:18:55.753382] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:18:55.793511] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:18:55.794175] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.output'),
                                  'state': {'begin': 1666743645.6223195,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 8 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.114681482315063,
                                            'end': 1666743656.737001,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_92.state')},
                       'cmd_93': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.output'),
                                  'state': {'begin': 1666743656.7398794,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7825148105621338,
                                            'end': 1666743657.5223942,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_93.state')},
                       'cmd_94': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '620874.40 IOPS     303.16 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '620874.40 IOPS     303.16 MiB/s\n'
                                            '[2022-10-26 04:19:08.547339] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:19:08.547409] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:19:08.547421] bdevperf '
                                            '[2022-10-26 04:19:08.547430] '
                                            '--no-shconf [2022-10-26 '
                                            '04:19:08.547438] -l 0 [2022-10-26 '
                                            '04:19:08.547446] --huge-unlink '
                                            '[2022-10-26 04:19:08.547454] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:19:08.547461] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:19:08.547469] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:19:08.547476] --iova-mode=pa '
                                            '[2022-10-26 04:19:08.547484] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:19:08.547492] '
                                            '--match-allocations [2022-10-26 '
                                            '04:19:08.547499] '
                                            '--file-prefix=spdk_pid138319 '
                                            '[2022-10-26 04:19:08.547506] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:19:08.685968] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:19:08.729521] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:19:08.730182] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.output'),
                                  'state': {'begin': 1666743658.5476117,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 8 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.11674427986145,
                                            'end': 1666743669.664356,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_94.state')},
                       'cmd_95': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.output'),
                                  'state': {'begin': 1666743669.666656,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7816565036773682,
                                            'end': 1666743670.4483125,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_95.state')},
                       'cmd_96': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '618850.10 IOPS     302.17 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '618850.10 IOPS     302.17 MiB/s\n'
                                            '[2022-10-26 04:19:21.485327] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:19:21.485385] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:19:21.485394] bdevperf '
                                            '[2022-10-26 04:19:21.485400] '
                                            '--no-shconf [2022-10-26 '
                                            '04:19:21.485406] -l 0 [2022-10-26 '
                                            '04:19:21.485411] --huge-unlink '
                                            '[2022-10-26 04:19:21.485418] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:19:21.485423] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:19:21.485429] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:19:21.485434] --iova-mode=pa '
                                            '[2022-10-26 04:19:21.485440] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:19:21.485445] '
                                            '--match-allocations [2022-10-26 '
                                            '04:19:21.485451] '
                                            '--file-prefix=spdk_pid138345 '
                                            '[2022-10-26 04:19:21.485456] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:19:21.617444] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:19:21.673483] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:19:21.674157] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.output'),
                                  'state': {'begin': 1666743671.4852216,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 8 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.12417483329773,
                                            'end': 1666743682.6093965,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_96.state')},
                       'cmd_97': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.output'),
                                  'state': {'begin': 1666743682.611976,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7869632244110107,
                                            'end': 1666743683.3989391,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_97.state')},
                       'cmd_98': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '518690.13 IOPS     253.27 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '518690.13 IOPS     253.27 MiB/s\n'
                                            '[2022-10-26 04:19:34.430663] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-26 04:19:34.430733] [ '
                                            'DPDK EAL parameters: [2022-10-26 '
                                            '04:19:34.430744] bdevperf '
                                            '[2022-10-26 04:19:34.430753] '
                                            '--no-shconf [2022-10-26 '
                                            '04:19:34.430761] -l 0 [2022-10-26 '
                                            '04:19:34.430769] --huge-unlink '
                                            '[2022-10-26 04:19:34.430776] '
                                            '--log-level=lib.eal:6 [2022-10-26 '
                                            '04:19:34.430783] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-26 04:19:34.430790] '
                                            '--log-level=user1:6 [2022-10-26 '
                                            '04:19:34.430798] --iova-mode=pa '
                                            '[2022-10-26 04:19:34.430805] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-26 04:19:34.430812] '
                                            '--match-allocations [2022-10-26 '
                                            '04:19:34.430819] '
                                            '--file-prefix=spdk_pid138371 '
                                            '[2022-10-26 04:19:34.430826] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-26 04:19:34.570480] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-26 04:19:34.614037] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-26 04:19:34.614728] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.output'),
                                  'state': {'begin': 1666743684.4286978,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 8 -o 512 -w randread -t '
                                                   "10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.109190702438354,
                                            'end': 1666743695.5378885,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_98.state')},
                       'cmd_99': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.output'),
                                  'state': {'begin': 1666743695.5408466,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7842028141021729,
                                            'end': 1666743696.3250494,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/001_bdevperf/cmd_99.state')}}},
 'id': '001_bdevperf',
 'name': 'bdevperf',
 'nr': 1,
 'status': {'elapsed': 777.7638144493103,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'bdevperf',
 'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['512'], 'repetitions': 3}}
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  171834.77 IOPS      83.90 MiB/s

 =============================================================

 Total                       :  171834.77 IOPS      83.90 MiB/s
[2022-10-26 04:09:12.283172] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:09:12.283242] [ DPDK EAL parameters: [2022-10-26 04:09:12.283253] bdevperf [2022-10-26 04:09:12.283263] --no-shconf [2022-10-26 04:09:12.283271] -l 0 [2022-10-26 04:09:12.283280] --huge-unlink [2022-10-26 04:09:12.283289] --log-level=lib.eal:6 [2022-10-26 04:09:12.283296] --log-level=lib.cryptodev:5 [2022-10-26 04:09:12.283304] --log-level=user1:6 [2022-10-26 04:09:12.283312] --iova-mode=pa [2022-10-26 04:09:12.283320] --base-virtaddr=0x200000000000 [2022-10-26 04:09:12.283328] --match-allocations [2022-10-26 04:09:12.283335] --file-prefix=spdk_pid137152 [2022-10-26 04:09:12.283344] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:09:12.421601] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:09:12.473509] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:09:12.474184] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  172633.05 IOPS      84.29 MiB/s

 =============================================================

 Total                       :  172633.05 IOPS      84.29 MiB/s
[2022-10-26 04:09:25.202810] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:09:25.202878] [ DPDK EAL parameters: [2022-10-26 04:09:25.202890] bdevperf [2022-10-26 04:09:25.202898] --no-shconf [2022-10-26 04:09:25.202906] -l 0 [2022-10-26 04:09:25.202914] --huge-unlink [2022-10-26 04:09:25.202921] --log-level=lib.eal:6 [2022-10-26 04:09:25.202930] --log-level=lib.cryptodev:5 [2022-10-26 04:09:25.202937] --log-level=user1:6 [2022-10-26 04:09:25.202944] --iova-mode=pa [2022-10-26 04:09:25.202952] --base-virtaddr=0x200000000000 [2022-10-26 04:09:25.202959] --match-allocations [2022-10-26 04:09:25.202967] --file-prefix=spdk_pid137177 [2022-10-26 04:09:25.202974] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:09:25.341687] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:09:25.389529] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:09:25.390195] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  171552.75 IOPS      83.77 MiB/s

 =============================================================

 Total                       :  171552.75 IOPS      83.77 MiB/s
[2022-10-26 04:09:38.130730] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:09:38.130800] [ DPDK EAL parameters: [2022-10-26 04:09:38.130811] bdevperf [2022-10-26 04:09:38.130819] --no-shconf [2022-10-26 04:09:38.130828] -l 0 [2022-10-26 04:09:38.130836] --huge-unlink [2022-10-26 04:09:38.130845] --log-level=lib.eal:6 [2022-10-26 04:09:38.130853] --log-level=lib.cryptodev:5 [2022-10-26 04:09:38.130860] --log-level=user1:6 [2022-10-26 04:09:38.130868] --iova-mode=pa [2022-10-26 04:09:38.130876] --base-virtaddr=0x200000000000 [2022-10-26 04:09:38.130885] --match-allocations [2022-10-26 04:09:38.130893] --file-prefix=spdk_pid137202 [2022-10-26 04:09:38.130901] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:09:38.269317] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:09:38.321522] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:09:38.322192] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  129651.27 IOPS      63.31 MiB/s

 =============================================================

 Total                       :  129651.27 IOPS      63.31 MiB/s
[2022-10-26 04:09:51.072421] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:09:51.072495] [ DPDK EAL parameters: [2022-10-26 04:09:51.072506] bdevperf [2022-10-26 04:09:51.072515] --no-shconf [2022-10-26 04:09:51.072524] -l 0 [2022-10-26 04:09:51.072532] --huge-unlink [2022-10-26 04:09:51.072541] --log-level=lib.eal:6 [2022-10-26 04:09:51.072549] --log-level=lib.cryptodev:5 [2022-10-26 04:09:51.072557] --log-level=user1:6 [2022-10-26 04:09:51.072566] --iova-mode=pa [2022-10-26 04:09:51.072574] --base-virtaddr=0x200000000000 [2022-10-26 04:09:51.072583] --match-allocations [2022-10-26 04:09:51.072591] --file-prefix=spdk_pid137229 [2022-10-26 04:09:51.072600] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:09:51.210930] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:09:51.257547] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:09:51.258212] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  129366.45 IOPS      63.17 MiB/s

 =============================================================

 Total                       :  129366.45 IOPS      63.17 MiB/s
[2022-10-26 04:10:03.980551] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:10:03.980620] [ DPDK EAL parameters: [2022-10-26 04:10:03.980631] bdevperf [2022-10-26 04:10:03.980640] --no-shconf [2022-10-26 04:10:03.980648] -l 0 [2022-10-26 04:10:03.980655] --huge-unlink [2022-10-26 04:10:03.980663] --log-level=lib.eal:6 [2022-10-26 04:10:03.980671] --log-level=lib.cryptodev:5 [2022-10-26 04:10:03.980678] --log-level=user1:6 [2022-10-26 04:10:03.980686] --iova-mode=pa [2022-10-26 04:10:03.980693] --base-virtaddr=0x200000000000 [2022-10-26 04:10:03.980700] --match-allocations [2022-10-26 04:10:03.980708] --file-prefix=spdk_pid137254 [2022-10-26 04:10:03.980716] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:10:04.119553] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:10:04.161522] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:10:04.162184] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  519280.46 IOPS     253.55 MiB/s

 =============================================================

 Total                       :  519280.46 IOPS     253.55 MiB/s
[2022-10-26 04:19:47.360269] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:19:47.360339] [ DPDK EAL parameters: [2022-10-26 04:19:47.360350] bdevperf [2022-10-26 04:19:47.360359] --no-shconf [2022-10-26 04:19:47.360367] -l 0 [2022-10-26 04:19:47.360375] --huge-unlink [2022-10-26 04:19:47.360384] --log-level=lib.eal:6 [2022-10-26 04:19:47.360392] --log-level=lib.cryptodev:5 [2022-10-26 04:19:47.360399] --log-level=user1:6 [2022-10-26 04:19:47.360407] --iova-mode=pa [2022-10-26 04:19:47.360416] --base-virtaddr=0x200000000000 [2022-10-26 04:19:47.360423] --match-allocations [2022-10-26 04:19:47.360431] --file-prefix=spdk_pid138396 [2022-10-26 04:19:47.360440] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:19:47.498685] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:19:47.549512] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:19:47.550170] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  520408.18 IOPS     254.11 MiB/s

 =============================================================

 Total                       :  520408.18 IOPS     254.11 MiB/s
[2022-10-26 04:20:00.275836] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:20:00.275907] [ DPDK EAL parameters: [2022-10-26 04:20:00.275918] bdevperf [2022-10-26 04:20:00.275926] --no-shconf [2022-10-26 04:20:00.275934] -l 0 [2022-10-26 04:20:00.275942] --huge-unlink [2022-10-26 04:20:00.275949] --log-level=lib.eal:6 [2022-10-26 04:20:00.275957] --log-level=lib.cryptodev:5 [2022-10-26 04:20:00.275964] --log-level=user1:6 [2022-10-26 04:20:00.275972] --iova-mode=pa [2022-10-26 04:20:00.275979] --base-virtaddr=0x200000000000 [2022-10-26 04:20:00.275987] --match-allocations [2022-10-26 04:20:00.275994] --file-prefix=spdk_pid138421 [2022-10-26 04:20:00.276001] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:20:00.414472] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:20:00.453507] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:20:00.454165] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  534233.37 IOPS     260.86 MiB/s

 =============================================================

 Total                       :  534233.37 IOPS     260.86 MiB/s
[2022-10-26 04:20:13.190404] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:20:13.190474] [ DPDK EAL parameters: [2022-10-26 04:20:13.190484] bdevperf [2022-10-26 04:20:13.190494] --no-shconf [2022-10-26 04:20:13.190502] -l 0 [2022-10-26 04:20:13.190509] --huge-unlink [2022-10-26 04:20:13.190517] --log-level=lib.eal:6 [2022-10-26 04:20:13.190525] --log-level=lib.cryptodev:5 [2022-10-26 04:20:13.190532] --log-level=user1:6 [2022-10-26 04:20:13.190539] --iova-mode=pa [2022-10-26 04:20:13.190546] --base-virtaddr=0x200000000000 [2022-10-26 04:20:13.190554] --match-allocations [2022-10-26 04:20:13.190561] --file-prefix=spdk_pid138447 [2022-10-26 04:20:13.190568] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:20:13.328990] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:20:13.369543] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:20:13.370218] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  527043.53 IOPS     257.35 MiB/s

 =============================================================

 Total                       :  527043.53 IOPS     257.35 MiB/s
[2022-10-26 04:20:26.114969] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:20:26.115039] [ DPDK EAL parameters: [2022-10-26 04:20:26.115050] bdevperf [2022-10-26 04:20:26.115059] --no-shconf [2022-10-26 04:20:26.115067] -l 0 [2022-10-26 04:20:26.115075] --huge-unlink [2022-10-26 04:20:26.115083] --log-level=lib.eal:6 [2022-10-26 04:20:26.115090] --log-level=lib.cryptodev:5 [2022-10-26 04:20:26.115097] --log-level=user1:6 [2022-10-26 04:20:26.115105] --iova-mode=pa [2022-10-26 04:20:26.115123] --base-virtaddr=0x200000000000 [2022-10-26 04:20:26.115131] --match-allocations [2022-10-26 04:20:26.115138] --file-prefix=spdk_pid138472 [2022-10-26 04:20:26.115146] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:20:26.253677] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:20:26.301556] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:20:26.302227] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  530831.57 IOPS     259.20 MiB/s

 =============================================================

 Total                       :  530831.57 IOPS     259.20 MiB/s
[2022-10-26 04:20:39.084853] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:20:39.084921] [ DPDK EAL parameters: [2022-10-26 04:20:39.084931] bdevperf [2022-10-26 04:20:39.084940] --no-shconf [2022-10-26 04:20:39.084949] -l 0 [2022-10-26 04:20:39.084957] --huge-unlink [2022-10-26 04:20:39.084966] --log-level=lib.eal:6 [2022-10-26 04:20:39.084974] --log-level=lib.cryptodev:5 [2022-10-26 04:20:39.084981] --log-level=user1:6 [2022-10-26 04:20:39.084990] --iova-mode=pa [2022-10-26 04:20:39.084998] --base-virtaddr=0x200000000000 [2022-10-26 04:20:39.085006] --match-allocations [2022-10-26 04:20:39.085015] --file-prefix=spdk_pid138497 [2022-10-26 04:20:39.085023] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:20:39.223403] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:20:39.269614] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:20:39.270283] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  533336.85 IOPS     260.42 MiB/s

 =============================================================

 Total                       :  533336.85 IOPS     260.42 MiB/s
[2022-10-26 04:20:52.015515] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:20:52.015584] [ DPDK EAL parameters: [2022-10-26 04:20:52.015595] bdevperf [2022-10-26 04:20:52.015604] --no-shconf [2022-10-26 04:20:52.015612] -l 0 [2022-10-26 04:20:52.015620] --huge-unlink [2022-10-26 04:20:52.015628] --log-level=lib.eal:6 [2022-10-26 04:20:52.015636] --log-level=lib.cryptodev:5 [2022-10-26 04:20:52.015643] --log-level=user1:6 [2022-10-26 04:20:52.015650] --iova-mode=pa [2022-10-26 04:20:52.015658] --base-virtaddr=0x200000000000 [2022-10-26 04:20:52.015665] --match-allocations [2022-10-26 04:20:52.015673] --file-prefix=spdk_pid138522 [2022-10-26 04:20:52.015680] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:20:52.154077] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:20:52.193585] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:20:52.194258] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  536250.58 IOPS     261.84 MiB/s

 =============================================================

 Total                       :  536250.58 IOPS     261.84 MiB/s
[2022-10-26 04:21:05.002701] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:21:05.002771] [ DPDK EAL parameters: [2022-10-26 04:21:05.002782] bdevperf [2022-10-26 04:21:05.002791] --no-shconf [2022-10-26 04:21:05.002800] -l 0 [2022-10-26 04:21:05.002808] --huge-unlink [2022-10-26 04:21:05.002816] --log-level=lib.eal:6 [2022-10-26 04:21:05.002823] --log-level=lib.cryptodev:5 [2022-10-26 04:21:05.002831] --log-level=user1:6 [2022-10-26 04:21:05.002846] --iova-mode=pa [2022-10-26 04:21:05.002854] --base-virtaddr=0x200000000000 [2022-10-26 04:21:05.002861] --match-allocations [2022-10-26 04:21:05.002870] --file-prefix=spdk_pid138547 [2022-10-26 04:21:05.002878] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:21:05.141323] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:21:05.185509] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:21:05.186190] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  534521.72 IOPS     261.00 MiB/s

 =============================================================

 Total                       :  534521.72 IOPS     261.00 MiB/s
[2022-10-26 04:21:17.969960] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:21:17.970030] [ DPDK EAL parameters: [2022-10-26 04:21:17.970041] bdevperf [2022-10-26 04:21:17.970050] --no-shconf [2022-10-26 04:21:17.970058] -l 0 [2022-10-26 04:21:17.970066] --huge-unlink [2022-10-26 04:21:17.970074] --log-level=lib.eal:6 [2022-10-26 04:21:17.970081] --log-level=lib.cryptodev:5 [2022-10-26 04:21:17.970088] --log-level=user1:6 [2022-10-26 04:21:17.970096] --iova-mode=pa [2022-10-26 04:21:17.970103] --base-virtaddr=0x200000000000 [2022-10-26 04:21:17.970111] --match-allocations [2022-10-26 04:21:17.970118] --file-prefix=spdk_pid138572 [2022-10-26 04:21:17.970126] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:21:18.108770] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:21:18.149565] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:21:18.150241] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  510627.58 IOPS     249.33 MiB/s

 =============================================================

 Total                       :  510627.58 IOPS     249.33 MiB/s
[2022-10-26 04:21:30.943091] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:21:30.943175] [ DPDK EAL parameters: [2022-10-26 04:21:30.943186] bdevperf [2022-10-26 04:21:30.943195] --no-shconf [2022-10-26 04:21:30.943204] -l 0 [2022-10-26 04:21:30.943211] --huge-unlink [2022-10-26 04:21:30.943220] --log-level=lib.eal:6 [2022-10-26 04:21:30.943227] --log-level=lib.cryptodev:5 [2022-10-26 04:21:30.943235] --log-level=user1:6 [2022-10-26 04:21:30.943243] --iova-mode=pa [2022-10-26 04:21:30.943251] --base-virtaddr=0x200000000000 [2022-10-26 04:21:30.943259] --match-allocations [2022-10-26 04:21:30.943267] --file-prefix=spdk_pid138597 [2022-10-26 04:21:30.943275] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:21:31.081602] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:21:31.121496] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:21:31.122161] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  507405.68 IOPS     247.76 MiB/s

 =============================================================

 Total                       :  507405.68 IOPS     247.76 MiB/s
[2022-10-26 04:21:43.951587] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:21:43.951660] [ DPDK EAL parameters: [2022-10-26 04:21:43.951671] bdevperf [2022-10-26 04:21:43.951680] --no-shconf [2022-10-26 04:21:43.951688] -l 0 [2022-10-26 04:21:43.951696] --huge-unlink [2022-10-26 04:21:43.951704] --log-level=lib.eal:6 [2022-10-26 04:21:43.951712] --log-level=lib.cryptodev:5 [2022-10-26 04:21:43.951719] --log-level=user1:6 [2022-10-26 04:21:43.951726] --iova-mode=pa [2022-10-26 04:21:43.951734] --base-virtaddr=0x200000000000 [2022-10-26 04:21:43.951741] --match-allocations [2022-10-26 04:21:43.951749] --file-prefix=spdk_pid138623 [2022-10-26 04:21:43.951756] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:21:44.090207] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:21:44.129577] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:21:44.130245] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  129066.85 IOPS      63.02 MiB/s

 =============================================================

 Total                       :  129066.85 IOPS      63.02 MiB/s
[2022-10-26 04:10:16.900959] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:10:16.901031] [ DPDK EAL parameters: [2022-10-26 04:10:16.901042] bdevperf [2022-10-26 04:10:16.901050] --no-shconf [2022-10-26 04:10:16.901058] -l 0 [2022-10-26 04:10:16.901066] --huge-unlink [2022-10-26 04:10:16.901074] --log-level=lib.eal:6 [2022-10-26 04:10:16.901081] --log-level=lib.cryptodev:5 [2022-10-26 04:10:16.901088] --log-level=user1:6 [2022-10-26 04:10:16.901096] --iova-mode=pa [2022-10-26 04:10:16.901104] --base-virtaddr=0x200000000000 [2022-10-26 04:10:16.901111] --match-allocations [2022-10-26 04:10:16.901118] --file-prefix=spdk_pid137279 [2022-10-26 04:10:16.901126] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:10:17.039698] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:10:17.081578] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:10:17.082253] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  510139.78 IOPS     249.09 MiB/s

 =============================================================

 Total                       :  510139.78 IOPS     249.09 MiB/s
[2022-10-26 04:21:56.963424] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:21:56.963490] [ DPDK EAL parameters: [2022-10-26 04:21:56.963502] bdevperf [2022-10-26 04:21:56.963511] --no-shconf [2022-10-26 04:21:56.963520] -l 0 [2022-10-26 04:21:56.963527] --huge-unlink [2022-10-26 04:21:56.963536] --log-level=lib.eal:6 [2022-10-26 04:21:56.963543] --log-level=lib.cryptodev:5 [2022-10-26 04:21:56.963551] --log-level=user1:6 [2022-10-26 04:21:56.963559] --iova-mode=pa [2022-10-26 04:21:56.963567] --base-virtaddr=0x200000000000 [2022-10-26 04:21:56.963575] --match-allocations [2022-10-26 04:21:56.963583] --file-prefix=spdk_pid138648 [2022-10-26 04:21:56.963590] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:21:57.102363] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:21:57.145569] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:21:57.146250] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  147294.48 IOPS      71.92 MiB/s

 =============================================================

 Total                       :  147294.48 IOPS      71.92 MiB/s
[2022-10-26 04:10:29.810550] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:10:29.810618] [ DPDK EAL parameters: [2022-10-26 04:10:29.810628] bdevperf [2022-10-26 04:10:29.810637] --no-shconf [2022-10-26 04:10:29.810646] -l 0 [2022-10-26 04:10:29.810653] --huge-unlink [2022-10-26 04:10:29.810662] --log-level=lib.eal:6 [2022-10-26 04:10:29.810670] --log-level=lib.cryptodev:5 [2022-10-26 04:10:29.810677] --log-level=user1:6 [2022-10-26 04:10:29.810685] --iova-mode=pa [2022-10-26 04:10:29.810694] --base-virtaddr=0x200000000000 [2022-10-26 04:10:29.810701] --match-allocations [2022-10-26 04:10:29.810710] --file-prefix=spdk_pid137304 [2022-10-26 04:10:29.810718] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:10:29.949133] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:10:29.993590] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:10:29.994247] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  147811.38 IOPS      72.17 MiB/s

 =============================================================

 Total                       :  147811.38 IOPS      72.17 MiB/s
[2022-10-26 04:10:42.746033] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:10:42.746103] [ DPDK EAL parameters: [2022-10-26 04:10:42.746113] bdevperf [2022-10-26 04:10:42.746122] --no-shconf [2022-10-26 04:10:42.746130] -l 0 [2022-10-26 04:10:42.746139] --huge-unlink [2022-10-26 04:10:42.746146] --log-level=lib.eal:6 [2022-10-26 04:10:42.746154] --log-level=lib.cryptodev:5 [2022-10-26 04:10:42.746161] --log-level=user1:6 [2022-10-26 04:10:42.746169] --iova-mode=pa [2022-10-26 04:10:42.746176] --base-virtaddr=0x200000000000 [2022-10-26 04:10:42.746183] --match-allocations [2022-10-26 04:10:42.746191] --file-prefix=spdk_pid137329 [2022-10-26 04:10:42.746198] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:10:42.884755] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:10:42.934563] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:10:42.935300] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  147836.09 IOPS      72.19 MiB/s

 =============================================================

 Total                       :  147836.09 IOPS      72.19 MiB/s
[2022-10-26 04:10:55.660577] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:10:55.660666] [ DPDK EAL parameters: [2022-10-26 04:10:55.660678] bdevperf [2022-10-26 04:10:55.660688] --no-shconf [2022-10-26 04:10:55.660697] -l 0 [2022-10-26 04:10:55.660706] --huge-unlink [2022-10-26 04:10:55.660716] --log-level=lib.eal:6 [2022-10-26 04:10:55.660724] --log-level=lib.cryptodev:5 [2022-10-26 04:10:55.660733] --log-level=user1:6 [2022-10-26 04:10:55.660742] --iova-mode=pa [2022-10-26 04:10:55.660750] --base-virtaddr=0x200000000000 [2022-10-26 04:10:55.660759] --match-allocations [2022-10-26 04:10:55.660768] --file-prefix=spdk_pid137354 [2022-10-26 04:10:55.660776] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:10:55.802085] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:10:55.849530] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:10:55.850223] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  145829.56 IOPS      71.21 MiB/s

 =============================================================

 Total                       :  145829.56 IOPS      71.21 MiB/s
[2022-10-26 04:11:08.598776] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:11:08.598845] [ DPDK EAL parameters: [2022-10-26 04:11:08.598856] bdevperf [2022-10-26 04:11:08.598865] --no-shconf [2022-10-26 04:11:08.598874] -l 0 [2022-10-26 04:11:08.598882] --huge-unlink [2022-10-26 04:11:08.598891] --log-level=lib.eal:6 [2022-10-26 04:11:08.598898] --log-level=lib.cryptodev:5 [2022-10-26 04:11:08.598905] --log-level=user1:6 [2022-10-26 04:11:08.598915] --iova-mode=pa [2022-10-26 04:11:08.598923] --base-virtaddr=0x200000000000 [2022-10-26 04:11:08.598931] --match-allocations [2022-10-26 04:11:08.598939] --file-prefix=spdk_pid137380 [2022-10-26 04:11:08.598947] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:11:08.737667] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:11:08.785519] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:11:08.786203] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  157191.61 IOPS      76.75 MiB/s

 =============================================================

 Total                       :  157191.61 IOPS      76.75 MiB/s
[2022-10-26 04:11:21.593539] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:11:21.593610] [ DPDK EAL parameters: [2022-10-26 04:11:21.593621] bdevperf [2022-10-26 04:11:21.593629] --no-shconf [2022-10-26 04:11:21.593638] -l 0 [2022-10-26 04:11:21.593646] --huge-unlink [2022-10-26 04:11:21.593653] --log-level=lib.eal:6 [2022-10-26 04:11:21.593661] --log-level=lib.cryptodev:5 [2022-10-26 04:11:21.593669] --log-level=user1:6 [2022-10-26 04:11:21.593677] --iova-mode=pa [2022-10-26 04:11:21.593685] --base-virtaddr=0x200000000000 [2022-10-26 04:11:21.593693] --match-allocations [2022-10-26 04:11:21.593701] --file-prefix=spdk_pid137406 [2022-10-26 04:11:21.593709] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:11:21.732218] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:11:21.777567] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:11:21.778256] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  145943.30 IOPS      71.26 MiB/s

 =============================================================

 Total                       :  145943.30 IOPS      71.26 MiB/s
[2022-10-26 04:11:34.584729] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:11:34.584800] [ DPDK EAL parameters: [2022-10-26 04:11:34.584811] bdevperf [2022-10-26 04:11:34.584820] --no-shconf [2022-10-26 04:11:34.584829] -l 0 [2022-10-26 04:11:34.584837] --huge-unlink [2022-10-26 04:11:34.584846] --log-level=lib.eal:6 [2022-10-26 04:11:34.584854] --log-level=lib.cryptodev:5 [2022-10-26 04:11:34.584861] --log-level=user1:6 [2022-10-26 04:11:34.584869] --iova-mode=pa [2022-10-26 04:11:34.584877] --base-virtaddr=0x200000000000 [2022-10-26 04:11:34.584885] --match-allocations [2022-10-26 04:11:34.584893] --file-prefix=spdk_pid137431 [2022-10-26 04:11:34.584902] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:11:34.723185] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:11:34.765492] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:11:34.766168] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  151211.44 IOPS      73.83 MiB/s

 =============================================================

 Total                       :  151211.44 IOPS      73.83 MiB/s
[2022-10-26 04:11:47.573241] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:11:47.573309] [ DPDK EAL parameters: [2022-10-26 04:11:47.573320] bdevperf [2022-10-26 04:11:47.573329] --no-shconf [2022-10-26 04:11:47.573338] -l 0 [2022-10-26 04:11:47.573346] --huge-unlink [2022-10-26 04:11:47.573355] --log-level=lib.eal:6 [2022-10-26 04:11:47.573363] --log-level=lib.cryptodev:5 [2022-10-26 04:11:47.573389] --log-level=user1:6 [2022-10-26 04:11:47.573398] --iova-mode=pa [2022-10-26 04:11:47.573406] --base-virtaddr=0x200000000000 [2022-10-26 04:11:47.573414] --match-allocations [2022-10-26 04:11:47.573426] --file-prefix=spdk_pid137456 [2022-10-26 04:11:47.573434] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:11:47.711808] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:11:47.753532] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:11:47.754194] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  139449.94 IOPS      68.09 MiB/s

 =============================================================

 Total                       :  139449.94 IOPS      68.09 MiB/s
[2022-10-26 04:12:00.586747] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:12:00.586818] [ DPDK EAL parameters: [2022-10-26 04:12:00.586829] bdevperf [2022-10-26 04:12:00.586837] --no-shconf [2022-10-26 04:12:00.586845] -l 0 [2022-10-26 04:12:00.586853] --huge-unlink [2022-10-26 04:12:00.586860] --log-level=lib.eal:6 [2022-10-26 04:12:00.586868] --log-level=lib.cryptodev:5 [2022-10-26 04:12:00.586875] --log-level=user1:6 [2022-10-26 04:12:00.586883] --iova-mode=pa [2022-10-26 04:12:00.586890] --base-virtaddr=0x200000000000 [2022-10-26 04:12:00.586898] --match-allocations [2022-10-26 04:12:00.586905] --file-prefix=spdk_pid137481 [2022-10-26 04:12:00.586912] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:12:00.725651] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:12:00.765524] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:12:00.766197] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  139549.74 IOPS      68.14 MiB/s

 =============================================================

 Total                       :  139549.74 IOPS      68.14 MiB/s
[2022-10-26 04:12:13.587573] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:12:13.587648] [ DPDK EAL parameters: [2022-10-26 04:12:13.587659] bdevperf [2022-10-26 04:12:13.587669] --no-shconf [2022-10-26 04:12:13.587678] -l 0 [2022-10-26 04:12:13.587686] --huge-unlink [2022-10-26 04:12:13.587695] --log-level=lib.eal:6 [2022-10-26 04:12:13.587703] --log-level=lib.cryptodev:5 [2022-10-26 04:12:13.587711] --log-level=user1:6 [2022-10-26 04:12:13.587720] --iova-mode=pa [2022-10-26 04:12:13.587728] --base-virtaddr=0x200000000000 [2022-10-26 04:12:13.587736] --match-allocations [2022-10-26 04:12:13.587744] --file-prefix=spdk_pid137506 [2022-10-26 04:12:13.587752] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:12:13.726489] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:12:13.773484] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:12:13.774158] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  262928.44 IOPS     128.38 MiB/s

 =============================================================

 Total                       :  262928.44 IOPS     128.38 MiB/s
[2022-10-26 04:12:26.587184] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:12:26.587258] [ DPDK EAL parameters: [2022-10-26 04:12:26.587270] bdevperf [2022-10-26 04:12:26.587278] --no-shconf [2022-10-26 04:12:26.587287] -l 0 [2022-10-26 04:12:26.587295] --huge-unlink [2022-10-26 04:12:26.587304] --log-level=lib.eal:6 [2022-10-26 04:12:26.587312] --log-level=lib.cryptodev:5 [2022-10-26 04:12:26.587319] --log-level=user1:6 [2022-10-26 04:12:26.587328] --iova-mode=pa [2022-10-26 04:12:26.587337] --base-virtaddr=0x200000000000 [2022-10-26 04:12:26.587344] --match-allocations [2022-10-26 04:12:26.587352] --file-prefix=spdk_pid137531 [2022-10-26 04:12:26.587360] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:12:26.725751] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:12:26.765496] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:12:26.766167] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  242424.58 IOPS     118.37 MiB/s

 =============================================================

 Total                       :  242424.58 IOPS     118.37 MiB/s
[2022-10-26 04:12:39.856534] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:12:39.856602] [ DPDK EAL parameters: [2022-10-26 04:12:39.856613] bdevperf [2022-10-26 04:12:39.856621] --no-shconf [2022-10-26 04:12:39.856629] -l 0 [2022-10-26 04:12:39.856637] --huge-unlink [2022-10-26 04:12:39.856644] --log-level=lib.eal:6 [2022-10-26 04:12:39.856652] --log-level=lib.cryptodev:5 [2022-10-26 04:12:39.856659] --log-level=user1:6 [2022-10-26 04:12:39.856667] --iova-mode=pa [2022-10-26 04:12:39.856674] --base-virtaddr=0x200000000000 [2022-10-26 04:12:39.856681] --match-allocations [2022-10-26 04:12:39.856688] --file-prefix=spdk_pid137556 [2022-10-26 04:12:39.856696] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:12:39.994898] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:12:40.041502] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:12:40.042162] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  263549.12 IOPS     128.69 MiB/s

 =============================================================

 Total                       :  263549.12 IOPS     128.69 MiB/s
[2022-10-26 04:12:52.784402] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:12:52.784475] [ DPDK EAL parameters: [2022-10-26 04:12:52.784486] bdevperf [2022-10-26 04:12:52.784494] --no-shconf [2022-10-26 04:12:52.784503] -l 0 [2022-10-26 04:12:52.784511] --huge-unlink [2022-10-26 04:12:52.784520] --log-level=lib.eal:6 [2022-10-26 04:12:52.784527] --log-level=lib.cryptodev:5 [2022-10-26 04:12:52.784536] --log-level=user1:6 [2022-10-26 04:12:52.784544] --iova-mode=pa [2022-10-26 04:12:52.784552] --base-virtaddr=0x200000000000 [2022-10-26 04:12:52.784560] --match-allocations [2022-10-26 04:12:52.784568] --file-prefix=spdk_pid137581 [2022-10-26 04:12:52.784576] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:12:52.923433] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:12:52.969615] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:12:52.970299] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  228150.59 IOPS     111.40 MiB/s

 =============================================================

 Total                       :  228150.59 IOPS     111.40 MiB/s
[2022-10-26 04:13:05.723004] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:13:05.723072] [ DPDK EAL parameters: [2022-10-26 04:13:05.723082] bdevperf [2022-10-26 04:13:05.723091] --no-shconf [2022-10-26 04:13:05.723100] -l 0 [2022-10-26 04:13:05.723108] --huge-unlink [2022-10-26 04:13:05.723127] --log-level=lib.eal:6 [2022-10-26 04:13:05.723136] --log-level=lib.cryptodev:5 [2022-10-26 04:13:05.723143] --log-level=user1:6 [2022-10-26 04:13:05.723151] --iova-mode=pa [2022-10-26 04:13:05.723159] --base-virtaddr=0x200000000000 [2022-10-26 04:13:05.723167] --match-allocations [2022-10-26 04:13:05.723176] --file-prefix=spdk_pid137607 [2022-10-26 04:13:05.723184] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:13:05.862734] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:13:05.917790] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:13:05.918471] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  230398.16 IOPS     112.50 MiB/s

 =============================================================

 Total                       :  230398.16 IOPS     112.50 MiB/s
[2022-10-26 04:13:18.651302] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:13:18.651371] [ DPDK EAL parameters: [2022-10-26 04:13:18.651381] bdevperf [2022-10-26 04:13:18.651390] --no-shconf [2022-10-26 04:13:18.651399] -l 0 [2022-10-26 04:13:18.651407] --huge-unlink [2022-10-26 04:13:18.651416] --log-level=lib.eal:6 [2022-10-26 04:13:18.651423] --log-level=lib.cryptodev:5 [2022-10-26 04:13:18.651431] --log-level=user1:6 [2022-10-26 04:13:18.651440] --iova-mode=pa [2022-10-26 04:13:18.651448] --base-virtaddr=0x200000000000 [2022-10-26 04:13:18.651456] --match-allocations [2022-10-26 04:13:18.651464] --file-prefix=spdk_pid137632 [2022-10-26 04:13:18.651472] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:13:18.790232] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:13:18.829548] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:13:18.830238] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  229316.75 IOPS     111.97 MiB/s

 =============================================================

 Total                       :  229316.75 IOPS     111.97 MiB/s
[2022-10-26 04:13:31.605298] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:13:31.605366] [ DPDK EAL parameters: [2022-10-26 04:13:31.605377] bdevperf [2022-10-26 04:13:31.605386] --no-shconf [2022-10-26 04:13:31.605394] -l 0 [2022-10-26 04:13:31.605402] --huge-unlink [2022-10-26 04:13:31.605411] --log-level=lib.eal:6 [2022-10-26 04:13:31.605419] --log-level=lib.cryptodev:5 [2022-10-26 04:13:31.605426] --log-level=user1:6 [2022-10-26 04:13:31.605434] --iova-mode=pa [2022-10-26 04:13:31.605441] --base-virtaddr=0x200000000000 [2022-10-26 04:13:31.605449] --match-allocations [2022-10-26 04:13:31.605456] --file-prefix=spdk_pid137657 [2022-10-26 04:13:31.605464] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:13:31.743848] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:13:31.793535] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:13:31.794215] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  291240.87 IOPS     142.21 MiB/s

 =============================================================

 Total                       :  291240.87 IOPS     142.21 MiB/s
[2022-10-26 04:13:44.534674] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:13:44.534744] [ DPDK EAL parameters: [2022-10-26 04:13:44.534755] bdevperf [2022-10-26 04:13:44.534764] --no-shconf [2022-10-26 04:13:44.534773] -l 0 [2022-10-26 04:13:44.534780] --huge-unlink [2022-10-26 04:13:44.534789] --log-level=lib.eal:6 [2022-10-26 04:13:44.534797] --log-level=lib.cryptodev:5 [2022-10-26 04:13:44.534805] --log-level=user1:6 [2022-10-26 04:13:44.534819] --iova-mode=pa [2022-10-26 04:13:44.534827] --base-virtaddr=0x200000000000 [2022-10-26 04:13:44.534835] --match-allocations [2022-10-26 04:13:44.534842] --file-prefix=spdk_pid137682 [2022-10-26 04:13:44.534850] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:13:44.673093] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:13:44.725564] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:13:44.726241] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  290441.54 IOPS     141.82 MiB/s

 =============================================================

 Total                       :  290441.54 IOPS     141.82 MiB/s
[2022-10-26 04:13:57.478245] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:13:57.478314] [ DPDK EAL parameters: [2022-10-26 04:13:57.478324] bdevperf [2022-10-26 04:13:57.478333] --no-shconf [2022-10-26 04:13:57.478341] -l 0 [2022-10-26 04:13:57.478349] --huge-unlink [2022-10-26 04:13:57.478357] --log-level=lib.eal:6 [2022-10-26 04:13:57.478365] --log-level=lib.cryptodev:5 [2022-10-26 04:13:57.478372] --log-level=user1:6 [2022-10-26 04:13:57.478380] --iova-mode=pa [2022-10-26 04:13:57.478387] --base-virtaddr=0x200000000000 [2022-10-26 04:13:57.478395] --match-allocations [2022-10-26 04:13:57.478402] --file-prefix=spdk_pid137708 [2022-10-26 04:13:57.478409] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:13:57.617477] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:13:57.661885] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:13:57.662595] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  290486.27 IOPS     141.84 MiB/s

 =============================================================

 Total                       :  290486.27 IOPS     141.84 MiB/s
[2022-10-26 04:14:10.393432] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:14:10.393501] [ DPDK EAL parameters: [2022-10-26 04:14:10.393511] bdevperf [2022-10-26 04:14:10.393520] --no-shconf [2022-10-26 04:14:10.393528] -l 0 [2022-10-26 04:14:10.393536] --huge-unlink [2022-10-26 04:14:10.393544] --log-level=lib.eal:6 [2022-10-26 04:14:10.393552] --log-level=lib.cryptodev:5 [2022-10-26 04:14:10.393559] --log-level=user1:6 [2022-10-26 04:14:10.393567] --iova-mode=pa [2022-10-26 04:14:10.393575] --base-virtaddr=0x200000000000 [2022-10-26 04:14:10.393583] --match-allocations [2022-10-26 04:14:10.393590] --file-prefix=spdk_pid137735 [2022-10-26 04:14:10.393598] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:14:10.531814] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:14:10.581554] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:14:10.582236] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  285675.37 IOPS     139.49 MiB/s

 =============================================================

 Total                       :  285675.37 IOPS     139.49 MiB/s
[2022-10-26 04:14:23.317220] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:14:23.317292] [ DPDK EAL parameters: [2022-10-26 04:14:23.317303] bdevperf [2022-10-26 04:14:23.317312] --no-shconf [2022-10-26 04:14:23.317320] -l 0 [2022-10-26 04:14:23.317329] --huge-unlink [2022-10-26 04:14:23.317338] --log-level=lib.eal:6 [2022-10-26 04:14:23.317345] --log-level=lib.cryptodev:5 [2022-10-26 04:14:23.317352] --log-level=user1:6 [2022-10-26 04:14:23.317360] --iova-mode=pa [2022-10-26 04:14:23.317367] --base-virtaddr=0x200000000000 [2022-10-26 04:14:23.317375] --match-allocations [2022-10-26 04:14:23.317383] --file-prefix=spdk_pid137760 [2022-10-26 04:14:23.317391] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:14:23.456191] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:14:23.509543] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:14:23.510240] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  285787.73 IOPS     139.54 MiB/s

 =============================================================

 Total                       :  285787.73 IOPS     139.54 MiB/s
[2022-10-26 04:14:36.284758] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:14:36.284827] [ DPDK EAL parameters: [2022-10-26 04:14:36.284838] bdevperf [2022-10-26 04:14:36.284847] --no-shconf [2022-10-26 04:14:36.284855] -l 0 [2022-10-26 04:14:36.284864] --huge-unlink [2022-10-26 04:14:36.284873] --log-level=lib.eal:6 [2022-10-26 04:14:36.284880] --log-level=lib.cryptodev:5 [2022-10-26 04:14:36.284888] --log-level=user1:6 [2022-10-26 04:14:36.284896] --iova-mode=pa [2022-10-26 04:14:36.284905] --base-virtaddr=0x200000000000 [2022-10-26 04:14:36.284913] --match-allocations [2022-10-26 04:14:36.284921] --file-prefix=spdk_pid137786 [2022-10-26 04:14:36.284929] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:14:36.423316] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:14:36.465562] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:14:36.466223] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  283717.83 IOPS     138.53 MiB/s

 =============================================================

 Total                       :  283717.83 IOPS     138.53 MiB/s
[2022-10-26 04:14:49.262571] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:14:49.262640] [ DPDK EAL parameters: [2022-10-26 04:14:49.262651] bdevperf [2022-10-26 04:14:49.262661] --no-shconf [2022-10-26 04:14:49.262669] -l 0 [2022-10-26 04:14:49.262676] --huge-unlink [2022-10-26 04:14:49.262684] --log-level=lib.eal:6 [2022-10-26 04:14:49.262692] --log-level=lib.cryptodev:5 [2022-10-26 04:14:49.262699] --log-level=user1:6 [2022-10-26 04:14:49.262706] --iova-mode=pa [2022-10-26 04:14:49.262714] --base-virtaddr=0x200000000000 [2022-10-26 04:14:49.262722] --match-allocations [2022-10-26 04:14:49.262729] --file-prefix=spdk_pid137811 [2022-10-26 04:14:49.262737] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:14:49.401169] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:14:49.449515] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:14:49.450200] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  273786.89 IOPS     133.69 MiB/s

 =============================================================

 Total                       :  273786.89 IOPS     133.69 MiB/s
[2022-10-26 04:15:02.256385] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:15:02.256453] [ DPDK EAL parameters: [2022-10-26 04:15:02.256464] bdevperf [2022-10-26 04:15:02.256473] --no-shconf [2022-10-26 04:15:02.256481] -l 0 [2022-10-26 04:15:02.256489] --huge-unlink [2022-10-26 04:15:02.256498] --log-level=lib.eal:6 [2022-10-26 04:15:02.256506] --log-level=lib.cryptodev:5 [2022-10-26 04:15:02.256514] --log-level=user1:6 [2022-10-26 04:15:02.256522] --iova-mode=pa [2022-10-26 04:15:02.256531] --base-virtaddr=0x200000000000 [2022-10-26 04:15:02.256539] --match-allocations [2022-10-26 04:15:02.256548] --file-prefix=spdk_pid137838 [2022-10-26 04:15:02.256556] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:15:02.395084] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:15:02.453546] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:15:02.454235] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  273253.44 IOPS     133.42 MiB/s

 =============================================================

 Total                       :  273253.44 IOPS     133.42 MiB/s
[2022-10-26 04:15:15.273977] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:15:15.274047] [ DPDK EAL parameters: [2022-10-26 04:15:15.274058] bdevperf [2022-10-26 04:15:15.274067] --no-shconf [2022-10-26 04:15:15.274075] -l 0 [2022-10-26 04:15:15.274082] --huge-unlink [2022-10-26 04:15:15.274091] --log-level=lib.eal:6 [2022-10-26 04:15:15.274098] --log-level=lib.cryptodev:5 [2022-10-26 04:15:15.274105] --log-level=user1:6 [2022-10-26 04:15:15.274120] --iova-mode=pa [2022-10-26 04:15:15.274128] --base-virtaddr=0x200000000000 [2022-10-26 04:15:15.274135] --match-allocations [2022-10-26 04:15:15.274142] --file-prefix=spdk_pid137864 [2022-10-26 04:15:15.274150] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:15:15.412303] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:15:15.453532] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:15:15.454217] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  273765.54 IOPS     133.67 MiB/s

 =============================================================

 Total                       :  273765.54 IOPS     133.67 MiB/s
[2022-10-26 04:15:28.271983] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:15:28.272054] [ DPDK EAL parameters: [2022-10-26 04:15:28.272065] bdevperf [2022-10-26 04:15:28.272073] --no-shconf [2022-10-26 04:15:28.272081] -l 0 [2022-10-26 04:15:28.272089] --huge-unlink [2022-10-26 04:15:28.272098] --log-level=lib.eal:6 [2022-10-26 04:15:28.272106] --log-level=lib.cryptodev:5 [2022-10-26 04:15:28.272114] --log-level=user1:6 [2022-10-26 04:15:28.272122] --iova-mode=pa [2022-10-26 04:15:28.272130] --base-virtaddr=0x200000000000 [2022-10-26 04:15:28.272137] --match-allocations [2022-10-26 04:15:28.272145] --file-prefix=spdk_pid137889 [2022-10-26 04:15:28.272153] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:15:28.410856] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:15:28.457568] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:15:28.458231] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  392406.39 IOPS     191.60 MiB/s

 =============================================================

 Total                       :  392406.39 IOPS     191.60 MiB/s
[2022-10-26 04:15:41.256135] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:15:41.256203] [ DPDK EAL parameters: [2022-10-26 04:15:41.256213] bdevperf [2022-10-26 04:15:41.256222] --no-shconf [2022-10-26 04:15:41.256230] -l 0 [2022-10-26 04:15:41.256238] --huge-unlink [2022-10-26 04:15:41.256246] --log-level=lib.eal:6 [2022-10-26 04:15:41.256253] --log-level=lib.cryptodev:5 [2022-10-26 04:15:41.256260] --log-level=user1:6 [2022-10-26 04:15:41.256268] --iova-mode=pa [2022-10-26 04:15:41.256276] --base-virtaddr=0x200000000000 [2022-10-26 04:15:41.256283] --match-allocations [2022-10-26 04:15:41.256290] --file-prefix=spdk_pid137914 [2022-10-26 04:15:41.256298] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:15:41.394453] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:15:41.441506] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:15:41.442191] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  392406.89 IOPS     191.60 MiB/s

 =============================================================

 Total                       :  392406.89 IOPS     191.60 MiB/s
[2022-10-26 04:15:54.215274] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:15:54.215345] [ DPDK EAL parameters: [2022-10-26 04:15:54.215356] bdevperf [2022-10-26 04:15:54.215365] --no-shconf [2022-10-26 04:15:54.215374] -l 0 [2022-10-26 04:15:54.215382] --huge-unlink [2022-10-26 04:15:54.215390] --log-level=lib.eal:6 [2022-10-26 04:15:54.215398] --log-level=lib.cryptodev:5 [2022-10-26 04:15:54.215406] --log-level=user1:6 [2022-10-26 04:15:54.215414] --iova-mode=pa [2022-10-26 04:15:54.215423] --base-virtaddr=0x200000000000 [2022-10-26 04:15:54.215431] --match-allocations [2022-10-26 04:15:54.215439] --file-prefix=spdk_pid137939 [2022-10-26 04:15:54.215448] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:15:54.353784] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:15:54.405744] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:15:54.406405] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  392071.55 IOPS     191.44 MiB/s

 =============================================================

 Total                       :  392071.55 IOPS     191.44 MiB/s
[2022-10-26 04:16:07.163390] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:16:07.163460] [ DPDK EAL parameters: [2022-10-26 04:16:07.163471] bdevperf [2022-10-26 04:16:07.163480] --no-shconf [2022-10-26 04:16:07.163488] -l 0 [2022-10-26 04:16:07.163496] --huge-unlink [2022-10-26 04:16:07.163503] --log-level=lib.eal:6 [2022-10-26 04:16:07.163511] --log-level=lib.cryptodev:5 [2022-10-26 04:16:07.163518] --log-level=user1:6 [2022-10-26 04:16:07.163526] --iova-mode=pa [2022-10-26 04:16:07.163533] --base-virtaddr=0x200000000000 [2022-10-26 04:16:07.163541] --match-allocations [2022-10-26 04:16:07.163548] --file-prefix=spdk_pid137964 [2022-10-26 04:16:07.163555] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:16:07.302310] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:16:07.345541] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:16:07.346210] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  430379.28 IOPS     210.15 MiB/s

 =============================================================

 Total                       :  430379.28 IOPS     210.15 MiB/s
[2022-10-26 04:16:20.087240] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:16:20.087312] [ DPDK EAL parameters: [2022-10-26 04:16:20.087323] bdevperf [2022-10-26 04:16:20.087332] --no-shconf [2022-10-26 04:16:20.087340] -l 0 [2022-10-26 04:16:20.087348] --huge-unlink [2022-10-26 04:16:20.087357] --log-level=lib.eal:6 [2022-10-26 04:16:20.087364] --log-level=lib.cryptodev:5 [2022-10-26 04:16:20.087372] --log-level=user1:6 [2022-10-26 04:16:20.087380] --iova-mode=pa [2022-10-26 04:16:20.087388] --base-virtaddr=0x200000000000 [2022-10-26 04:16:20.087396] --match-allocations [2022-10-26 04:16:20.087404] --file-prefix=spdk_pid137989 [2022-10-26 04:16:20.087413] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:16:20.225818] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:16:20.269590] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:16:20.270271] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  431736.81 IOPS     210.81 MiB/s

 =============================================================

 Total                       :  431736.81 IOPS     210.81 MiB/s
[2022-10-26 04:16:33.002825] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:16:33.002895] [ DPDK EAL parameters: [2022-10-26 04:16:33.002905] bdevperf [2022-10-26 04:16:33.002915] --no-shconf [2022-10-26 04:16:33.002923] -l 0 [2022-10-26 04:16:33.002930] --huge-unlink [2022-10-26 04:16:33.002938] --log-level=lib.eal:6 [2022-10-26 04:16:33.002946] --log-level=lib.cryptodev:5 [2022-10-26 04:16:33.002954] --log-level=user1:6 [2022-10-26 04:16:33.002962] --iova-mode=pa [2022-10-26 04:16:33.002970] --base-virtaddr=0x200000000000 [2022-10-26 04:16:33.002977] --match-allocations [2022-10-26 04:16:33.002985] --file-prefix=spdk_pid138014 [2022-10-26 04:16:33.002992] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:16:33.141476] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:16:33.189533] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:16:33.190204] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  428711.29 IOPS     209.33 MiB/s

 =============================================================

 Total                       :  428711.29 IOPS     209.33 MiB/s
[2022-10-26 04:16:45.923337] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:16:45.923406] [ DPDK EAL parameters: [2022-10-26 04:16:45.923416] bdevperf [2022-10-26 04:16:45.923425] --no-shconf [2022-10-26 04:16:45.923433] -l 0 [2022-10-26 04:16:45.923441] --huge-unlink [2022-10-26 04:16:45.923449] --log-level=lib.eal:6 [2022-10-26 04:16:45.923456] --log-level=lib.cryptodev:5 [2022-10-26 04:16:45.923464] --log-level=user1:6 [2022-10-26 04:16:45.923471] --iova-mode=pa [2022-10-26 04:16:45.923479] --base-virtaddr=0x200000000000 [2022-10-26 04:16:45.923486] --match-allocations [2022-10-26 04:16:45.923494] --file-prefix=spdk_pid138039 [2022-10-26 04:16:45.923501] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:16:46.062040] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:16:46.101512] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:16:46.102194] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  485865.81 IOPS     237.24 MiB/s

 =============================================================

 Total                       :  485865.81 IOPS     237.24 MiB/s
[2022-10-26 04:16:58.841718] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:16:58.841789] [ DPDK EAL parameters: [2022-10-26 04:16:58.841800] bdevperf [2022-10-26 04:16:58.841809] --no-shconf [2022-10-26 04:16:58.841817] -l 0 [2022-10-26 04:16:58.841825] --huge-unlink [2022-10-26 04:16:58.841833] --log-level=lib.eal:6 [2022-10-26 04:16:58.841840] --log-level=lib.cryptodev:5 [2022-10-26 04:16:58.841848] --log-level=user1:6 [2022-10-26 04:16:58.841855] --iova-mode=pa [2022-10-26 04:16:58.841863] --base-virtaddr=0x200000000000 [2022-10-26 04:16:58.841870] --match-allocations [2022-10-26 04:16:58.841878] --file-prefix=spdk_pid138064 [2022-10-26 04:16:58.841885] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:16:58.980698] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:16:59.025507] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:16:59.026173] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  482781.41 IOPS     235.73 MiB/s

 =============================================================

 Total                       :  482781.41 IOPS     235.73 MiB/s
[2022-10-26 04:17:11.766933] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:17:11.767002] [ DPDK EAL parameters: [2022-10-26 04:17:11.767013] bdevperf [2022-10-26 04:17:11.767022] --no-shconf [2022-10-26 04:17:11.767030] -l 0 [2022-10-26 04:17:11.767038] --huge-unlink [2022-10-26 04:17:11.767047] --log-level=lib.eal:6 [2022-10-26 04:17:11.767055] --log-level=lib.cryptodev:5 [2022-10-26 04:17:11.767062] --log-level=user1:6 [2022-10-26 04:17:11.767071] --iova-mode=pa [2022-10-26 04:17:11.767079] --base-virtaddr=0x200000000000 [2022-10-26 04:17:11.767087] --match-allocations [2022-10-26 04:17:11.767095] --file-prefix=spdk_pid138092 [2022-10-26 04:17:11.767103] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:17:11.905315] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:17:11.945540] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:17:11.946218] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  482648.51 IOPS     235.67 MiB/s

 =============================================================

 Total                       :  482648.51 IOPS     235.67 MiB/s
[2022-10-26 04:17:24.672976] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:17:24.673047] [ DPDK EAL parameters: [2022-10-26 04:17:24.673057] bdevperf [2022-10-26 04:17:24.673066] --no-shconf [2022-10-26 04:17:24.673074] -l 0 [2022-10-26 04:17:24.673082] --huge-unlink [2022-10-26 04:17:24.673089] --log-level=lib.eal:6 [2022-10-26 04:17:24.673097] --log-level=lib.cryptodev:5 [2022-10-26 04:17:24.673104] --log-level=user1:6 [2022-10-26 04:17:24.673112] --iova-mode=pa [2022-10-26 04:17:24.673119] --base-virtaddr=0x200000000000 [2022-10-26 04:17:24.673126] --match-allocations [2022-10-26 04:17:24.673134] --file-prefix=spdk_pid138117 [2022-10-26 04:17:24.673141] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:17:24.812469] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:17:24.861970] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:17:24.862650] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  479021.22 IOPS     233.90 MiB/s

 =============================================================

 Total                       :  479021.22 IOPS     233.90 MiB/s
[2022-10-26 04:17:37.615480] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:17:37.615550] [ DPDK EAL parameters: [2022-10-26 04:17:37.615561] bdevperf [2022-10-26 04:17:37.615570] --no-shconf [2022-10-26 04:17:37.615578] -l 0 [2022-10-26 04:17:37.615586] --huge-unlink [2022-10-26 04:17:37.615594] --log-level=lib.eal:6 [2022-10-26 04:17:37.615601] --log-level=lib.cryptodev:5 [2022-10-26 04:17:37.615608] --log-level=user1:6 [2022-10-26 04:17:37.615616] --iova-mode=pa [2022-10-26 04:17:37.615623] --base-virtaddr=0x200000000000 [2022-10-26 04:17:37.615631] --match-allocations [2022-10-26 04:17:37.615639] --file-prefix=spdk_pid138142 [2022-10-26 04:17:37.615647] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:17:37.754205] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:17:37.797538] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:17:37.798220] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  479430.62 IOPS     234.10 MiB/s

 =============================================================

 Total                       :  479430.62 IOPS     234.10 MiB/s
[2022-10-26 04:17:50.615507] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:17:50.615575] [ DPDK EAL parameters: [2022-10-26 04:17:50.615586] bdevperf [2022-10-26 04:17:50.615594] --no-shconf [2022-10-26 04:17:50.615602] -l 0 [2022-10-26 04:17:50.615610] --huge-unlink [2022-10-26 04:17:50.615617] --log-level=lib.eal:6 [2022-10-26 04:17:50.615625] --log-level=lib.cryptodev:5 [2022-10-26 04:17:50.615632] --log-level=user1:6 [2022-10-26 04:17:50.615640] --iova-mode=pa [2022-10-26 04:17:50.615648] --base-virtaddr=0x200000000000 [2022-10-26 04:17:50.615655] --match-allocations [2022-10-26 04:17:50.615663] --file-prefix=spdk_pid138167 [2022-10-26 04:17:50.615670] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:17:50.752773] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:17:50.805554] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:17:50.806237] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  480153.62 IOPS     234.45 MiB/s

 =============================================================

 Total                       :  480153.62 IOPS     234.45 MiB/s
[2022-10-26 04:18:03.607723] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:18:03.607800] [ DPDK EAL parameters: [2022-10-26 04:18:03.607811] bdevperf [2022-10-26 04:18:03.607820] --no-shconf [2022-10-26 04:18:03.607828] -l 0 [2022-10-26 04:18:03.607836] --huge-unlink [2022-10-26 04:18:03.607844] --log-level=lib.eal:6 [2022-10-26 04:18:03.607852] --log-level=lib.cryptodev:5 [2022-10-26 04:18:03.607859] --log-level=user1:6 [2022-10-26 04:18:03.607868] --iova-mode=pa [2022-10-26 04:18:03.607876] --base-virtaddr=0x200000000000 [2022-10-26 04:18:03.607884] --match-allocations [2022-10-26 04:18:03.607892] --file-prefix=spdk_pid138192 [2022-10-26 04:18:03.607900] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:18:03.746385] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:18:03.789546] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:18:03.790227] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  430950.51 IOPS     210.43 MiB/s

 =============================================================

 Total                       :  430950.51 IOPS     210.43 MiB/s
[2022-10-26 04:18:16.597404] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:18:16.597472] [ DPDK EAL parameters: [2022-10-26 04:18:16.597483] bdevperf [2022-10-26 04:18:16.597491] --no-shconf [2022-10-26 04:18:16.597499] -l 0 [2022-10-26 04:18:16.597507] --huge-unlink [2022-10-26 04:18:16.597516] --log-level=lib.eal:6 [2022-10-26 04:18:16.597524] --log-level=lib.cryptodev:5 [2022-10-26 04:18:16.597531] --log-level=user1:6 [2022-10-26 04:18:16.597557] --iova-mode=pa [2022-10-26 04:18:16.597567] --base-virtaddr=0x200000000000 [2022-10-26 04:18:16.597575] --match-allocations [2022-10-26 04:18:16.597583] --file-prefix=spdk_pid138217 [2022-10-26 04:18:16.597599] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:18:16.735774] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:18:16.777548] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:18:16.778223] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  429672.51 IOPS     209.80 MiB/s

 =============================================================

 Total                       :  429672.51 IOPS     209.80 MiB/s
[2022-10-26 04:18:29.614303] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:18:29.614376] [ DPDK EAL parameters: [2022-10-26 04:18:29.614387] bdevperf [2022-10-26 04:18:29.614395] --no-shconf [2022-10-26 04:18:29.614403] -l 0 [2022-10-26 04:18:29.614411] --huge-unlink [2022-10-26 04:18:29.614419] --log-level=lib.eal:6 [2022-10-26 04:18:29.614426] --log-level=lib.cryptodev:5 [2022-10-26 04:18:29.614433] --log-level=user1:6 [2022-10-26 04:18:29.614441] --iova-mode=pa [2022-10-26 04:18:29.614448] --base-virtaddr=0x200000000000 [2022-10-26 04:18:29.614455] --match-allocations [2022-10-26 04:18:29.614463] --file-prefix=spdk_pid138243 [2022-10-26 04:18:29.614470] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:18:29.753633] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:18:29.801578] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:18:29.802241] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  430511.26 IOPS     210.21 MiB/s

 =============================================================

 Total                       :  430511.26 IOPS     210.21 MiB/s
[2022-10-26 04:18:42.616257] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:18:42.616325] [ DPDK EAL parameters: [2022-10-26 04:18:42.616336] bdevperf [2022-10-26 04:18:42.616345] --no-shconf [2022-10-26 04:18:42.616353] -l 0 [2022-10-26 04:18:42.616362] --huge-unlink [2022-10-26 04:18:42.616370] --log-level=lib.eal:6 [2022-10-26 04:18:42.616378] --log-level=lib.cryptodev:5 [2022-10-26 04:18:42.616386] --log-level=user1:6 [2022-10-26 04:18:42.616394] --iova-mode=pa [2022-10-26 04:18:42.616410] --base-virtaddr=0x200000000000 [2022-10-26 04:18:42.616419] --match-allocations [2022-10-26 04:18:42.616427] --file-prefix=spdk_pid138269 [2022-10-26 04:18:42.616435] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:18:42.754809] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:18:42.801477] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:18:42.802142] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  616823.81 IOPS     301.18 MiB/s

 =============================================================

 Total                       :  616823.81 IOPS     301.18 MiB/s
[2022-10-26 04:18:55.614634] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:18:55.614701] [ DPDK EAL parameters: [2022-10-26 04:18:55.614712] bdevperf [2022-10-26 04:18:55.614722] --no-shconf [2022-10-26 04:18:55.614730] -l 0 [2022-10-26 04:18:55.614737] --huge-unlink [2022-10-26 04:18:55.614745] --log-level=lib.eal:6 [2022-10-26 04:18:55.614752] --log-level=lib.cryptodev:5 [2022-10-26 04:18:55.614759] --log-level=user1:6 [2022-10-26 04:18:55.614767] --iova-mode=pa [2022-10-26 04:18:55.614774] --base-virtaddr=0x200000000000 [2022-10-26 04:18:55.614782] --match-allocations [2022-10-26 04:18:55.614789] --file-prefix=spdk_pid138294 [2022-10-26 04:18:55.614796] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:18:55.753382] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:18:55.793511] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:18:55.794175] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  620874.40 IOPS     303.16 MiB/s

 =============================================================

 Total                       :  620874.40 IOPS     303.16 MiB/s
[2022-10-26 04:19:08.547339] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:19:08.547409] [ DPDK EAL parameters: [2022-10-26 04:19:08.547421] bdevperf [2022-10-26 04:19:08.547430] --no-shconf [2022-10-26 04:19:08.547438] -l 0 [2022-10-26 04:19:08.547446] --huge-unlink [2022-10-26 04:19:08.547454] --log-level=lib.eal:6 [2022-10-26 04:19:08.547461] --log-level=lib.cryptodev:5 [2022-10-26 04:19:08.547469] --log-level=user1:6 [2022-10-26 04:19:08.547476] --iova-mode=pa [2022-10-26 04:19:08.547484] --base-virtaddr=0x200000000000 [2022-10-26 04:19:08.547492] --match-allocations [2022-10-26 04:19:08.547499] --file-prefix=spdk_pid138319 [2022-10-26 04:19:08.547506] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:19:08.685968] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:19:08.729521] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:19:08.730182] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  618850.10 IOPS     302.17 MiB/s

 =============================================================

 Total                       :  618850.10 IOPS     302.17 MiB/s
[2022-10-26 04:19:21.485327] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:19:21.485385] [ DPDK EAL parameters: [2022-10-26 04:19:21.485394] bdevperf [2022-10-26 04:19:21.485400] --no-shconf [2022-10-26 04:19:21.485406] -l 0 [2022-10-26 04:19:21.485411] --huge-unlink [2022-10-26 04:19:21.485418] --log-level=lib.eal:6 [2022-10-26 04:19:21.485423] --log-level=lib.cryptodev:5 [2022-10-26 04:19:21.485429] --log-level=user1:6 [2022-10-26 04:19:21.485434] --iova-mode=pa [2022-10-26 04:19:21.485440] --base-virtaddr=0x200000000000 [2022-10-26 04:19:21.485445] --match-allocations [2022-10-26 04:19:21.485451] --file-prefix=spdk_pid138345 [2022-10-26 04:19:21.485456] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:19:21.617444] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:19:21.673483] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:19:21.674157] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 512 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  518690.13 IOPS     253.27 MiB/s

 =============================================================

 Total                       :  518690.13 IOPS     253.27 MiB/s
[2022-10-26 04:19:34.430663] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-26 04:19:34.430733] [ DPDK EAL parameters: [2022-10-26 04:19:34.430744] bdevperf [2022-10-26 04:19:34.430753] --no-shconf [2022-10-26 04:19:34.430761] -l 0 [2022-10-26 04:19:34.430769] --huge-unlink [2022-10-26 04:19:34.430776] --log-level=lib.eal:6 [2022-10-26 04:19:34.430783] --log-level=lib.cryptodev:5 [2022-10-26 04:19:34.430790] --log-level=user1:6 [2022-10-26 04:19:34.430798] --iova-mode=pa [2022-10-26 04:19:34.430805] --base-virtaddr=0x200000000000 [2022-10-26 04:19:34.430812] --match-allocations [2022-10-26 04:19:34.430819] --file-prefix=spdk_pid138371 [2022-10-26 04:19:34.430826] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-26 04:19:34.570480] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-26 04:19:34.614037] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-26 04:19:34.614728] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Extract and transform fio and bdevperf output
    =============================================

    fio has:

    * bandwidth in KiB / second
    * latency is mean and in nano-seconds
    * IOPS are 'raw' / base-unit

    bdevperf has:

    * bandwidth in MiB / second
    * IOPS are 'raw' / base-unit

    NOTE: The metric-context need addition of backend-options, the options are
    semi-encoded by ioengine and 'label', however, that is not very precise.

{'description': '\n'
                '    Extract and transform fio and bdevperf output\n'
                '    =============================================\n'
                '\n'
                '    fio has:\n'
                '\n'
                '    * bandwidth in KiB / second\n'
                '    * latency is mean and in nano-seconds\n'
                "    * IOPS are 'raw' / base-unit\n"
                '\n'
                '    bdevperf has:\n'
                '\n'
                '    * bandwidth in MiB / second\n'
                "    * IOPS are 'raw' / base-unit\n"
                '\n'
                '    NOTE: The metric-context need addition of '
                'backend-options, the options are\n'
                "    semi-encoded by ioengine and 'label', however, that is "
                'not very precise.\n',
 'extras': {},
 'id': '002_plot_bdevperf',
 'name': 'plot_bdevperf',
 'nr': 2,
 'status': {'elapsed': 0.644026517868042,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'plot',
 'with': {'path': 'cijoe-output',
          'plot_attributes': 'plot-attributes-512.yaml',
          'tool': 'bdevperf'}}

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Run fio...

{'description': '\n    Run fio...\n',
 'extras': {'runlog': {'cmd_01': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.output'),
                                  'state': {'begin': 1666743838.8012276,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7809967994689941,
                                            'end': 1666743839.5822244,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_01.state')},
                       'cmd_02': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.output'),
                                  'state': {'begin': 1666743839.5842044,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring_cmd" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.04334831237793,
                                            'end': 1666743850.6275527,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_02.state')},
                       'cmd_03': {'output': 'io_uring_cmd-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring_cmd, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666750939,\n'
                                            '  "timestamp_ms" : '
                                            '1666750939831,\n'
                                            '  "time" : "Wed Oct 26 04:22:19 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"io_uring_cmd",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 298279936,\n'
                                            '        "io_kbytes" : 291289,\n'
                                            '        "bw_bytes" : 42605332,\n'
                                            '        "bw" : 41606,\n'
                                            '        "iops" : 83213.540923,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 582578,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2224,\n'
                                            '          "max" : 28664,\n'
                                            '          "mean" : 2408.062177,\n'
                                            '          "stddev" : 216.254296,\n'
                                            '          "N" : 582578\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 171,\n'
                                            '          "max" : 108203,\n'
                                            '          "mean" : 9002.665866,\n'
                                            '          "stddev" : '
                                            '1272.220914,\n'
                                            '          "N" : 582578,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8640,\n'
                                            '            "5.000000" : 8640,\n'
                                            '            "10.000000" : 8768,\n'
                                            '            "20.000000" : 8768,\n'
                                            '            "30.000000" : 8768,\n'
                                            '            "40.000000" : 8768,\n'
                                            '            "50.000000" : 8768,\n'
                                            '            "60.000000" : 8896,\n'
                                            '            "70.000000" : 8896,\n'
                                            '            "80.000000" : 8896,\n'
                                            '            "90.000000" : 9024,\n'
                                            '            "95.000000" : 9152,\n'
                                            '            "99.000000" : 15680,\n'
                                            '            "99.500000" : 16320,\n'
                                            '            "99.900000" : 19072,\n'
                                            '            "99.950000" : 21888,\n'
                                            '            "99.990000" : 55552\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 8035,\n'
                                            '          "max" : 110557,\n'
                                            '          "mean" : 11410.728043,\n'
                                            '          "stddev" : '
                                            '1289.083345,\n'
                                            '          "N" : 582578\n'
                                            '        },\n'
                                            '        "bw_min" : 41479,\n'
                                            '        "bw_max" : 41783,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '41617.071429,\n'
                                            '        "bw_dev" : 75.011245,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 82958,\n'
                                            '        "iops_max" : 83566,\n'
                                            '        "iops_mean" : '
                                            '83234.142857,\n'
                                            '        "iops_stddev" : '
                                            '150.022489,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 11.700000,\n'
                                            '      "sys_cpu" : 46.957143,\n'
                                            '      "ctx" : 582454,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 97.381638,\n'
                                            '        "20" : 2.541119,\n'
                                            '        "50" : 0.052868,\n'
                                            '        "100" : 0.013389,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-reference: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=138675: Wed Oct 26 04:22:19 '
                                            '2022\n'
                                            '  read: IOPS=83.2k, BW=40.6MiB/s '
                                            '(42.6MB/s)(284MiB/7001msec)\n'
                                            '    slat (nsec): min=2224, '
                                            'max=28664, avg=2408.06, '
                                            'stdev=216.25\n'
                                            '    clat (nsec): min=171, '
                                            'max=108203, avg=9002.67, '
                                            'stdev=1272.22\n'
                                            '     lat (usec): min=8, max=110, '
                                            'avg=11.41, stdev= 1.29\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8640],  5.00th=[ '
                                            '8640], 10.00th=[ 8768], 20.00th=[ '
                                            '8768],\n'
                                            '     | 30.00th=[ 8768], 40.00th=[ '
                                            '8768], 50.00th=[ 8768], 60.00th=[ '
                                            '8896],\n'
                                            '     | 70.00th=[ 8896], 80.00th=[ '
                                            '8896], 90.00th=[ 9024], 95.00th=[ '
                                            '9152],\n'
                                            '     | 99.00th=[15680], '
                                            '99.50th=[16320], 99.90th=[19072], '
                                            '99.95th=[21888],\n'
                                            '     | 99.99th=[55552]\n'
                                            '   bw (  KiB/s): min=41479, '
                                            'max=41783, per=100.00%, '
                                            'avg=41617.07, stdev=75.01, '
                                            'samples=14\n'
                                            '   iops        : min=82958, '
                                            'max=83566, avg=83234.14, '
                                            'stdev=150.02, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=97.38%, 20=2.54%, 50=0.05%, '
                                            '100=0.01%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=11.70%, '
                                            'sys=46.96%, ctx=582454, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=582578,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=40.6MiB/s (42.6MB/s), '
                                            '40.6MiB/s-40.6MiB/s '
                                            '(42.6MB/s-42.6MB/s), io=284MiB '
                                            '(298MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.output'),
                                  'state': {'begin': 1666743850.6302538,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7856242656707764,
                                            'end': 1666743851.415878,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_03.state')},
                       'cmd_04': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.output'),
                                  'state': {'begin': 1666743852.4110315,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.793879508972168,
                                            'end': 1666743853.204911,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_04.state')},
                       'cmd_05': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.output'),
                                  'state': {'begin': 1666743853.2070465,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring_cmd" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.047319650650024,
                                            'end': 1666743864.2543662,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_05.state')},
                       'cmd_06': {'output': 'io_uring_cmd-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring_cmd, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666750953,\n'
                                            '  "timestamp_ms" : '
                                            '1666750953459,\n'
                                            '  "time" : "Wed Oct 26 04:22:33 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"io_uring_cmd",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 299215872,\n'
                                            '        "io_kbytes" : 292203,\n'
                                            '        "bw_bytes" : 42739018,\n'
                                            '        "bw" : 41737,\n'
                                            '        "iops" : 83474.646479,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 584406,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2245,\n'
                                            '          "max" : 30237,\n'
                                            '          "mean" : 2410.540881,\n'
                                            '          "stddev" : 223.499942,\n'
                                            '          "N" : 584406\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 190,\n'
                                            '          "max" : 117380,\n'
                                            '          "mean" : 8968.711232,\n'
                                            '          "stddev" : '
                                            '1341.037263,\n'
                                            '          "N" : 584406,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8640,\n'
                                            '            "5.000000" : 8640,\n'
                                            '            "10.000000" : 8640,\n'
                                            '            "20.000000" : 8768,\n'
                                            '            "30.000000" : 8768,\n'
                                            '            "40.000000" : 8768,\n'
                                            '            "50.000000" : 8768,\n'
                                            '            "60.000000" : 8768,\n'
                                            '            "70.000000" : 8768,\n'
                                            '            "80.000000" : 8896,\n'
                                            '            "90.000000" : 8896,\n'
                                            '            "95.000000" : 9024,\n'
                                            '            "99.000000" : 15680,\n'
                                            '            "99.500000" : 16512,\n'
                                            '            "99.900000" : 19072,\n'
                                            '            "99.950000" : 22912,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 8446,\n'
                                            '          "max" : 119784,\n'
                                            '          "mean" : 11379.252112,\n'
                                            '          "stddev" : '
                                            '1357.748455,\n'
                                            '          "N" : 584406\n'
                                            '        },\n'
                                            '        "bw_min" : 41568,\n'
                                            '        "bw_max" : 41880,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '41747.214286,\n'
                                            '        "bw_dev" : 89.871856,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 83136,\n'
                                            '        "iops_max" : 83760,\n'
                                            '        "iops_mean" : '
                                            '83494.500000,\n'
                                            '        "iops_stddev" : '
                                            '179.757422,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 10.600000,\n'
                                            '      "sys_cpu" : 47.971429,\n'
                                            '      "ctx" : 584275,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 97.434147,\n'
                                            '        "20" : 2.479270,\n'
                                            '        "50" : 0.057152,\n'
                                            '        "100" : 0.018651,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-reference: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=138713: Wed Oct 26 04:22:33 '
                                            '2022\n'
                                            '  read: IOPS=83.5k, BW=40.8MiB/s '
                                            '(42.7MB/s)(285MiB/7001msec)\n'
                                            '    slat (nsec): min=2245, '
                                            'max=30237, avg=2410.54, '
                                            'stdev=223.50\n'
                                            '    clat (nsec): min=190, '
                                            'max=117380, avg=8968.71, '
                                            'stdev=1341.04\n'
                                            '     lat (usec): min=8, max=119, '
                                            'avg=11.38, stdev= 1.36\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8640],  5.00th=[ '
                                            '8640], 10.00th=[ 8640], 20.00th=[ '
                                            '8768],\n'
                                            '     | 30.00th=[ 8768], 40.00th=[ '
                                            '8768], 50.00th=[ 8768], 60.00th=[ '
                                            '8768],\n'
                                            '     | 70.00th=[ 8768], 80.00th=[ '
                                            '8896], 90.00th=[ 8896], 95.00th=[ '
                                            '9024],\n'
                                            '     | 99.00th=[15680], '
                                            '99.50th=[16512], 99.90th=[19072], '
                                            '99.95th=[22912],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=41568, '
                                            'max=41880, per=100.00%, '
                                            'avg=41747.21, stdev=89.87, '
                                            'samples=14\n'
                                            '   iops        : min=83136, '
                                            'max=83760, avg=83494.50, '
                                            'stdev=179.76, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 10=97.43%, '
                                            '20=2.48%, 50=0.06%, 100=0.02%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=10.60%, '
                                            'sys=47.97%, ctx=584275, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=584406,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=40.8MiB/s (42.7MB/s), '
                                            '40.8MiB/s-40.8MiB/s '
                                            '(42.7MB/s-42.7MB/s), io=285MiB '
                                            '(299MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.output'),
                                  'state': {'begin': 1666743864.2571094,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8469626903533936,
                                            'end': 1666743865.104072,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_06.state')},
                       'cmd_07': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.output'),
                                  'state': {'begin': 1666743866.22143,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8789708614349365,
                                            'end': 1666743867.100401,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_07.state')},
                       'cmd_08': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.output'),
                                  'state': {'begin': 1666743867.1025667,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring_cmd" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.065075635910034,
                                            'end': 1666743878.1676424,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_08.state')},
                       'cmd_09': {'output': 'io_uring_cmd-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring_cmd, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666750967,\n'
                                            '  "timestamp_ms" : '
                                            '1666750967372,\n'
                                            '  "time" : "Wed Oct 26 04:22:47 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"io_uring_cmd",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 299606016,\n'
                                            '        "io_kbytes" : 292584,\n'
                                            '        "bw_bytes" : 42794745,\n'
                                            '        "bw" : 41791,\n'
                                            '        "iops" : 83583.345236,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 585167,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2244,\n'
                                            '          "max" : 36078,\n'
                                            '          "mean" : 2458.829481,\n'
                                            '          "stddev" : 222.580918,\n'
                                            '          "N" : 585168\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 170,\n'
                                            '          "max" : 108393,\n'
                                            '          "mean" : 8930.150774,\n'
                                            '          "stddev" : '
                                            '1341.344004,\n'
                                            '          "N" : 585167,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8512,\n'
                                            '            "5.000000" : 8640,\n'
                                            '            "10.000000" : 8640,\n'
                                            '            "20.000000" : 8640,\n'
                                            '            "30.000000" : 8640,\n'
                                            '            "40.000000" : 8768,\n'
                                            '            "50.000000" : 8768,\n'
                                            '            "60.000000" : 8768,\n'
                                            '            "70.000000" : 8768,\n'
                                            '            "80.000000" : 8768,\n'
                                            '            "90.000000" : 8896,\n'
                                            '            "95.000000" : 9024,\n'
                                            '            "99.000000" : 15680,\n'
                                            '            "99.500000" : 16320,\n'
                                            '            "99.900000" : 19072,\n'
                                            '            "99.950000" : 22656,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 8526,\n'
                                            '          "max" : 110867,\n'
                                            '          "mean" : 11388.980330,\n'
                                            '          "stddev" : '
                                            '1357.766479,\n'
                                            '          "N" : 585167\n'
                                            '        },\n'
                                            '        "bw_min" : 41636,\n'
                                            '        "bw_max" : 41936,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '41802.000000,\n'
                                            '        "bw_dev" : 76.181968,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 83272,\n'
                                            '        "iops_max" : 83872,\n'
                                            '        "iops_mean" : '
                                            '83604.071429,\n'
                                            '        "iops_stddev" : '
                                            '152.343975,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 12.771429,\n'
                                            '      "sys_cpu" : 45.685714,\n'
                                            '      "ctx" : 585039,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 97.435775,\n'
                                            '        "20" : 2.483394,\n'
                                            '        "50" : 0.051097,\n'
                                            '        "100" : 0.018285,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-reference: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=138751: Wed Oct 26 04:22:47 '
                                            '2022\n'
                                            '  read: IOPS=83.6k, BW=40.8MiB/s '
                                            '(42.8MB/s)(286MiB/7001msec)\n'
                                            '    slat (nsec): min=2244, '
                                            'max=36078, avg=2458.83, '
                                            'stdev=222.58\n'
                                            '    clat (nsec): min=170, '
                                            'max=108393, avg=8930.15, '
                                            'stdev=1341.34\n'
                                            '     lat (usec): min=8, max=110, '
                                            'avg=11.39, stdev= 1.36\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8512],  5.00th=[ '
                                            '8640], 10.00th=[ 8640], 20.00th=[ '
                                            '8640],\n'
                                            '     | 30.00th=[ 8640], 40.00th=[ '
                                            '8768], 50.00th=[ 8768], 60.00th=[ '
                                            '8768],\n'
                                            '     | 70.00th=[ 8768], 80.00th=[ '
                                            '8768], 90.00th=[ 8896], 95.00th=[ '
                                            '9024],\n'
                                            '     | 99.00th=[15680], '
                                            '99.50th=[16320], 99.90th=[19072], '
                                            '99.95th=[22656],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=41636, '
                                            'max=41936, per=100.00%, '
                                            'avg=41802.00, stdev=76.18, '
                                            'samples=14\n'
                                            '   iops        : min=83272, '
                                            'max=83872, avg=83604.07, '
                                            'stdev=152.34, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 10=97.44%, '
                                            '20=2.48%, 50=0.05%, 100=0.02%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=12.77%, '
                                            'sys=45.69%, ctx=585039, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=585167,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=40.8MiB/s (42.8MB/s), '
                                            '40.8MiB/s-40.8MiB/s '
                                            '(42.8MB/s-42.8MB/s), io=286MiB '
                                            '(300MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.output'),
                                  'state': {'begin': 1666743878.1701941,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7935676574707031,
                                            'end': 1666743878.9637618,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_09.state')},
                       'cmd_10': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.output'),
                                  'state': {'begin': 1666743879.9947097,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7847332954406738,
                                            'end': 1666743880.779443,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_10.state')},
                       'cmd_100': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.output'),
                                   'state': {'begin': 1666744310.7295651,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7809097766876221,
                                             'end': 1666744311.510475,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_100.state')},
                       'cmd_101': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.output'),
                                   'state': {'begin': 1666744311.5130086,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.03992748260498,
                                             'end': 1666744322.552936,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_101.state')},
                       'cmd_102': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751411,\n'
                                             '  "timestamp_ms" : '
                                             '1666751411748,\n'
                                             '  "time" : "Wed Oct 26 04:30:11 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 737202688,\n'
                                             '        "io_kbytes" : 719924,\n'
                                             '        "bw_bytes" : 105299626,\n'
                                             '        "bw" : 102831,\n'
                                             '        "iops" : 205663.190973,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1439848,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1152,\n'
                                             '          "max" : 27401,\n'
                                             '          "mean" : 1324.465338,\n'
                                             '          "stddev" : '
                                             '150.281294,\n'
                                             '          "N" : 1439848\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 80,\n'
                                             '          "max" : 101000,\n'
                                             '          "mean" : 8115.300295,\n'
                                             '          "stddev" : '
                                             '2223.862930,\n'
                                             '          "N" : 1439849,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6048,\n'
                                             '            "5.000000" : 6112,\n'
                                             '            "10.000000" : 6112,\n'
                                             '            "20.000000" : 6176,\n'
                                             '            "30.000000" : 6240,\n'
                                             '            "40.000000" : 6304,\n'
                                             '            "50.000000" : 8768,\n'
                                             '            "60.000000" : 9536,\n'
                                             '            "70.000000" : 9536,\n'
                                             '            "80.000000" : 9664,\n'
                                             '            "90.000000" : 9664,\n'
                                             '            "95.000000" : 9792,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '17280,\n'
                                             '            "99.900000" : '
                                             '18560,\n'
                                             '            "99.950000" : '
                                             '22656,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5911,\n'
                                             '          "max" : 102202,\n'
                                             '          "mean" : 9439.765673,\n'
                                             '          "stddev" : '
                                             '2290.846561,\n'
                                             '          "N" : 1439849\n'
                                             '        },\n'
                                             '        "bw_min" : 102294,\n'
                                             '        "bw_max" : 103753,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '102856.357143,\n'
                                             '        "bw_dev" : 489.366562,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 204588,\n'
                                             '        "iops_max" : 207506,\n'
                                             '        "iops_mean" : '
                                             '205712.714286,\n'
                                             '        "iops_stddev" : '
                                             '978.733124,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 18.900000,\n'
                                             '      "sys_cpu" : 50.014286,\n'
                                             '      "ctx" : 719850,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.593182,\n'
                                             '        "20" : 3.315558,\n'
                                             '        "50" : 0.057784,\n'
                                             '        "100" : 0.023961,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=140132: Wed Oct 26 04:30:11 '
                                             '2022\n'
                                             '  read: IOPS=206k, BW=100MiB/s '
                                             '(105MB/s)(703MiB/7001msec)\n'
                                             '    slat (nsec): min=1152, '
                                             'max=27401, avg=1324.47, '
                                             'stdev=150.28\n'
                                             '    clat (nsec): min=80, '
                                             'max=101000, avg=8115.30, '
                                             'stdev=2223.86\n'
                                             '     lat (usec): min=5, max=102, '
                                             'avg= 9.44, stdev= 2.29\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6048],  '
                                             '5.00th=[ 6112], 10.00th=[ 6112], '
                                             '20.00th=[ 6176],\n'
                                             '     | 30.00th=[ 6240], '
                                             '40.00th=[ 6304], 50.00th=[ '
                                             '8768], 60.00th=[ 9536],\n'
                                             '     | 70.00th=[ 9536], '
                                             '80.00th=[ 9664], 90.00th=[ '
                                             '9664], 95.00th=[ 9792],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[17280], '
                                             '99.90th=[18560], '
                                             '99.95th=[22656],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=102294, '
                                             'max=103753, per=100.00%, '
                                             'avg=102856.36, stdev=489.37, '
                                             'samples=14\n'
                                             '   iops        : min=204588, '
                                             'max=207506, avg=205712.71, '
                                             'stdev=978.73, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=96.59%, 20=3.32%, '
                                             '50=0.06%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=18.90%, '
                                             'sys=50.01%, ctx=719850, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1439848,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=100MiB/s (105MB/s), '
                                             '100MiB/s-100MiB/s '
                                             '(105MB/s-105MB/s), io=703MiB '
                                             '(737MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.output'),
                                   'state': {'begin': 1666744322.5554903,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7864255905151367,
                                             'end': 1666744323.3419158,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_102.state')},
                       'cmd_103': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.output'),
                                   'state': {'begin': 1666744324.3314447,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7831816673278809,
                                             'end': 1666744325.1146264,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_103.state')},
                       'cmd_104': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.output'),
                                   'state': {'begin': 1666744325.1165314,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.035388708114624,
                                             'end': 1666744336.15192,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_104.state')},
                       'cmd_105': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751425,\n'
                                             '  "timestamp_ms" : '
                                             '1666751425348,\n'
                                             '  "time" : "Wed Oct 26 04:30:25 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 741398528,\n'
                                             '        "io_kbytes" : 724022,\n'
                                             '        "bw_bytes" : 105898947,\n'
                                             '        "bw" : 103416,\n'
                                             '        "iops" : 206833.738037,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1448043,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1142,\n'
                                             '          "max" : 40395,\n'
                                             '          "mean" : 1297.229181,\n'
                                             '          "stddev" : '
                                             '146.300890,\n'
                                             '          "N" : 1448043\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 80,\n'
                                             '          "max" : 96732,\n'
                                             '          "mean" : 8087.446961,\n'
                                             '          "stddev" : '
                                             '2216.390509,\n'
                                             '          "N" : 1448044,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5984,\n'
                                             '            "5.000000" : 6048,\n'
                                             '            "10.000000" : 6112,\n'
                                             '            "20.000000" : 6176,\n'
                                             '            "30.000000" : 6240,\n'
                                             '            "40.000000" : 6304,\n'
                                             '            "50.000000" : 8096,\n'
                                             '            "60.000000" : 9408,\n'
                                             '            "70.000000" : 9536,\n'
                                             '            "80.000000" : 9536,\n'
                                             '            "90.000000" : 9664,\n'
                                             '            "95.000000" : 9792,\n'
                                             '            "99.000000" : '
                                             '16320,\n'
                                             '            "99.500000" : '
                                             '17280,\n'
                                             '            "99.900000" : '
                                             '19072,\n'
                                             '            "99.950000" : '
                                             '26752,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6071,\n'
                                             '          "max" : 97904,\n'
                                             '          "mean" : 9384.676166,\n'
                                             '          "stddev" : '
                                             '2292.860984,\n'
                                             '          "N" : 1448044\n'
                                             '        },\n'
                                             '        "bw_min" : 103177,\n'
                                             '        "bw_max" : 103737,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '103442.142857,\n'
                                             '        "bw_dev" : 188.649801,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 206354,\n'
                                             '        "iops_max" : 207474,\n'
                                             '        "iops_mean" : '
                                             '206884.285714,\n'
                                             '        "iops_stddev" : '
                                             '377.119330,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.357143,\n'
                                             '      "sys_cpu" : 51.185714,\n'
                                             '      "ctx" : 723965,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.716948,\n'
                                             '        "20" : 3.182640,\n'
                                             '        "50" : 0.069128,\n'
                                             '        "100" : 0.023135,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=140171: Wed Oct 26 04:30:25 '
                                             '2022\n'
                                             '  read: IOPS=207k, BW=101MiB/s '
                                             '(106MB/s)(707MiB/7001msec)\n'
                                             '    slat (nsec): min=1142, '
                                             'max=40395, avg=1297.23, '
                                             'stdev=146.30\n'
                                             '    clat (nsec): min=80, '
                                             'max=96732, avg=8087.45, '
                                             'stdev=2216.39\n'
                                             '     lat (nsec): min=6071, '
                                             'max=97904, avg=9384.68, '
                                             'stdev=2292.86\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5984],  '
                                             '5.00th=[ 6048], 10.00th=[ 6112], '
                                             '20.00th=[ 6176],\n'
                                             '     | 30.00th=[ 6240], '
                                             '40.00th=[ 6304], 50.00th=[ '
                                             '8096], 60.00th=[ 9408],\n'
                                             '     | 70.00th=[ 9536], '
                                             '80.00th=[ 9536], 90.00th=[ '
                                             '9664], 95.00th=[ 9792],\n'
                                             '     | 99.00th=[16320], '
                                             '99.50th=[17280], '
                                             '99.90th=[19072], '
                                             '99.95th=[26752],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=103177, '
                                             'max=103737, per=100.00%, '
                                             'avg=103442.14, stdev=188.65, '
                                             'samples=14\n'
                                             '   iops        : min=206354, '
                                             'max=207474, avg=206884.29, '
                                             'stdev=377.12, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=96.72%, 20=3.18%, '
                                             '50=0.07%\n'
                                             '  lat (usec)   : 100=0.02%\n'
                                             '  cpu          : usr=17.36%, '
                                             'sys=51.19%, ctx=723965, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1448043,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=101MiB/s (106MB/s), '
                                             '101MiB/s-101MiB/s '
                                             '(106MB/s-106MB/s), io=707MiB '
                                             '(741MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.output'),
                                   'state': {'begin': 1666744336.1546147,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7847998142242432,
                                             'end': 1666744336.9394145,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_105.state')},
                       'cmd_106': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.output'),
                                   'state': {'begin': 1666744337.9332614,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7826714515686035,
                                             'end': 1666744338.7159328,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_106.state')},
                       'cmd_107': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.output'),
                                   'state': {'begin': 1666744338.7180254,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.03706669807434,
                                             'end': 1666744349.7550921,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_107.state')},
                       'cmd_108': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751438,\n'
                                             '  "timestamp_ms" : '
                                             '1666751438950,\n'
                                             '  "time" : "Wed Oct 26 04:30:38 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 734948864,\n'
                                             '        "io_kbytes" : 717723,\n'
                                             '        "bw_bytes" : 104977698,\n'
                                             '        "bw" : 102517,\n'
                                             '        "iops" : 205034.280817,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1435445,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1182,\n'
                                             '          "max" : 27221,\n'
                                             '          "mean" : 1330.533098,\n'
                                             '          "stddev" : '
                                             '132.496465,\n'
                                             '          "N" : 1435446\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 100,\n'
                                             '          "max" : 117019,\n'
                                             '          "mean" : 8150.587585,\n'
                                             '          "stddev" : '
                                             '2236.609171,\n'
                                             '          "N" : 1435446,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6048,\n'
                                             '            "5.000000" : 6112,\n'
                                             '            "10.000000" : 6112,\n'
                                             '            "20.000000" : 6240,\n'
                                             '            "30.000000" : 6304,\n'
                                             '            "40.000000" : 6304,\n'
                                             '            "50.000000" : 8640,\n'
                                             '            "60.000000" : 9536,\n'
                                             '            "70.000000" : 9664,\n'
                                             '            "80.000000" : 9664,\n'
                                             '            "90.000000" : 9792,\n'
                                             '            "95.000000" : 9792,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '17280,\n'
                                             '            "99.900000" : '
                                             '20096,\n'
                                             '            "99.950000" : '
                                             '27008,\n'
                                             '            "99.990000" : 62208\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5871,\n'
                                             '          "max" : 118241,\n'
                                             '          "mean" : 9481.120683,\n'
                                             '          "stddev" : '
                                             '2308.727929,\n'
                                             '          "N" : 1435446\n'
                                             '        },\n'
                                             '        "bw_min" : 102325,\n'
                                             '        "bw_max" : 102710,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '102550.214286,\n'
                                             '        "bw_dev" : 111.494173,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 204650,\n'
                                             '        "iops_max" : 205420,\n'
                                             '        "iops_mean" : '
                                             '205100.428571,\n'
                                             '        "iops_stddev" : '
                                             '223.226933,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.157143,\n'
                                             '      "sys_cpu" : 48.828571,\n'
                                             '      "ctx" : 717624,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.261577,\n'
                                             '        "20" : 3.628561,\n'
                                             '        "50" : 0.077885,\n'
                                             '        "100" : 0.021875,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=140209: Wed Oct 26 04:30:38 '
                                             '2022\n'
                                             '  read: IOPS=205k, BW=100MiB/s '
                                             '(105MB/s)(701MiB/7001msec)\n'
                                             '    slat (nsec): min=1182, '
                                             'max=27221, avg=1330.53, '
                                             'stdev=132.50\n'
                                             '    clat (nsec): min=100, '
                                             'max=117019, avg=8150.59, '
                                             'stdev=2236.61\n'
                                             '     lat (usec): min=5, max=118, '
                                             'avg= 9.48, stdev= 2.31\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6048],  '
                                             '5.00th=[ 6112], 10.00th=[ 6112], '
                                             '20.00th=[ 6240],\n'
                                             '     | 30.00th=[ 6304], '
                                             '40.00th=[ 6304], 50.00th=[ '
                                             '8640], 60.00th=[ 9536],\n'
                                             '     | 70.00th=[ 9664], '
                                             '80.00th=[ 9664], 90.00th=[ '
                                             '9792], 95.00th=[ 9792],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[17280], '
                                             '99.90th=[20096], '
                                             '99.95th=[27008],\n'
                                             '     | 99.99th=[62208]\n'
                                             '   bw (  KiB/s): min=102325, '
                                             'max=102710, per=100.00%, '
                                             'avg=102550.21, stdev=111.49, '
                                             'samples=14\n'
                                             '   iops        : min=204650, '
                                             'max=205420, avg=205100.43, '
                                             'stdev=223.23, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=96.26%, 20=3.63%, '
                                             '50=0.08%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=20.16%, '
                                             'sys=48.83%, ctx=717624, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1435445,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=100MiB/s (105MB/s), '
                                             '100MiB/s-100MiB/s '
                                             '(105MB/s-105MB/s), io=701MiB '
                                             '(735MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.output'),
                                   'state': {'begin': 1666744349.7578328,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7910897731781006,
                                             'end': 1666744350.5489225,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_108.state')},
                       'cmd_109': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.output'),
                                   'state': {'begin': 1666744351.544084,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7850728034973145,
                                             'end': 1666744352.3291569,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_109.state')},
                       'cmd_11': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.output'),
                                  'state': {'begin': 1666743880.7814655,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring_cmd" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.055826425552368,
                                            'end': 1666743891.837292,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_11.state')},
                       'cmd_110': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.output'),
                                   'state': {'begin': 1666744352.3311505,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.055224418640137,
                                             'end': 1666744363.386375,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_110.state')},
                       'cmd_111': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751452,\n'
                                             '  "timestamp_ms" : '
                                             '1666751452581,\n'
                                             '  "time" : "Wed Oct 26 04:30:52 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 726595072,\n'
                                             '        "io_kbytes" : 709565,\n'
                                             '        "bw_bytes" : 103784469,\n'
                                             '        "bw" : 101352,\n'
                                             '        "iops" : 202703.756606,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1419129,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1212,\n'
                                             '          "max" : 64080,\n'
                                             '          "mean" : 1388.671381,\n'
                                             '          "stddev" : '
                                             '173.652155,\n'
                                             '          "N" : 1419130\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 121,\n'
                                             '          "max" : 84979,\n'
                                             '          "mean" : 8206.406104,\n'
                                             '          "stddev" : '
                                             '2089.729687,\n'
                                             '          "N" : 1419130,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5216,\n'
                                             '            "5.000000" : 5536,\n'
                                             '            "10.000000" : 6304,\n'
                                             '            "20.000000" : 6368,\n'
                                             '            "30.000000" : 6496,\n'
                                             '            "40.000000" : 7776,\n'
                                             '            "50.000000" : 8096,\n'
                                             '            "60.000000" : 8256,\n'
                                             '            "70.000000" : 9792,\n'
                                             '            "80.000000" : 9920,\n'
                                             '            "90.000000" : '
                                             '10048,\n'
                                             '            "95.000000" : '
                                             '10176,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '17024,\n'
                                             '            "99.900000" : '
                                             '18560,\n'
                                             '            "99.950000" : '
                                             '22912,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5921,\n'
                                             '          "max" : 86271,\n'
                                             '          "mean" : 9595.077485,\n'
                                             '          "stddev" : '
                                             '2157.718644,\n'
                                             '          "N" : 1419130\n'
                                             '        },\n'
                                             '        "bw_min" : 101000,\n'
                                             '        "bw_max" : 101608,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '101386.571429,\n'
                                             '        "bw_dev" : 192.066380,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 202000,\n'
                                             '        "iops_max" : 203216,\n'
                                             '        "iops_mean" : '
                                             '202773.285714,\n'
                                             '        "iops_stddev" : '
                                             '384.198953,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.300000,\n'
                                             '      "sys_cpu" : 49.800000,\n'
                                             '      "ctx" : 709604,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 87.470413,\n'
                                             '        "20" : 12.438193,\n'
                                             '        "50" : 0.058275,\n'
                                             '        "100" : 0.019730,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=140247: Wed '
                                             'Oct 26 04:30:52 2022\n'
                                             '  read: IOPS=203k, BW=99.0MiB/s '
                                             '(104MB/s)(693MiB/7001msec)\n'
                                             '    slat (nsec): min=1212, '
                                             'max=64080, avg=1388.67, '
                                             'stdev=173.65\n'
                                             '    clat (nsec): min=121, '
                                             'max=84979, avg=8206.41, '
                                             'stdev=2089.73\n'
                                             '     lat (nsec): min=5921, '
                                             'max=86271, avg=9595.08, '
                                             'stdev=2157.72\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5216],  '
                                             '5.00th=[ 5536], 10.00th=[ 6304], '
                                             '20.00th=[ 6368],\n'
                                             '     | 30.00th=[ 6496], '
                                             '40.00th=[ 7776], 50.00th=[ '
                                             '8096], 60.00th=[ 8256],\n'
                                             '     | 70.00th=[ 9792], '
                                             '80.00th=[ 9920], '
                                             '90.00th=[10048], '
                                             '95.00th=[10176],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[17024], '
                                             '99.90th=[18560], '
                                             '99.95th=[22912],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=101000, '
                                             'max=101608, per=100.00%, '
                                             'avg=101386.57, stdev=192.07, '
                                             'samples=14\n'
                                             '   iops        : min=202000, '
                                             'max=203216, avg=202773.29, '
                                             'stdev=384.20, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=87.47%, 20=12.44%, '
                                             '50=0.06%\n'
                                             '  lat (usec)   : 100=0.02%\n'
                                             '  cpu          : usr=19.30%, '
                                             'sys=49.80%, ctx=709604, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1419129,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=99.0MiB/s (104MB/s), '
                                             '99.0MiB/s-99.0MiB/s '
                                             '(104MB/s-104MB/s), io=693MiB '
                                             '(727MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.output'),
                                   'state': {'begin': 1666744363.389406,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7918994426727295,
                                             'end': 1666744364.1813054,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_111.state')},
                       'cmd_112': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.output'),
                                   'state': {'begin': 1666744365.1733491,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7847476005554199,
                                             'end': 1666744365.9580967,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_112.state')},
                       'cmd_113': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.output'),
                                   'state': {'begin': 1666744365.9600933,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.058468341827393,
                                             'end': 1666744377.0185616,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_113.state')},
                       'cmd_114': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751466,\n'
                                             '  "timestamp_ms" : '
                                             '1666751466213,\n'
                                             '  "time" : "Wed Oct 26 04:31:06 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 722791424,\n'
                                             '        "io_kbytes" : 705851,\n'
                                             '        "bw_bytes" : 103241168,\n'
                                             '        "bw" : 100821,\n'
                                             '        "iops" : 201642.765319,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1411701,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1202,\n'
                                             '          "max" : 25348,\n'
                                             '          "mean" : 1368.416727,\n'
                                             '          "stddev" : '
                                             '146.449095,\n'
                                             '          "N" : 1411701\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 120,\n'
                                             '          "max" : 109946,\n'
                                             '          "mean" : 8258.915002,\n'
                                             '          "stddev" : '
                                             '2093.679574,\n'
                                             '          "N" : 1411702,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5472,\n'
                                             '            "5.000000" : 6240,\n'
                                             '            "10.000000" : 6240,\n'
                                             '            "20.000000" : 6304,\n'
                                             '            "30.000000" : 6432,\n'
                                             '            "40.000000" : 7776,\n'
                                             '            "50.000000" : 8032,\n'
                                             '            "60.000000" : 8256,\n'
                                             '            "70.000000" : 9792,\n'
                                             '            "80.000000" : 9920,\n'
                                             '            "90.000000" : 9920,\n'
                                             '            "95.000000" : '
                                             '10048,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '17024,\n'
                                             '            "99.900000" : '
                                             '18560,\n'
                                             '            "99.950000" : '
                                             '23680,\n'
                                             '            "99.990000" : 60672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6092,\n'
                                             '          "max" : 111208,\n'
                                             '          "mean" : 9627.331782,\n'
                                             '          "stddev" : '
                                             '2159.028715,\n'
                                             '          "N" : 1411702\n'
                                             '        },\n'
                                             '        "bw_min" : 100667,\n'
                                             '        "bw_max" : 101169,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '100850.000000,\n'
                                             '        "bw_dev" : 142.687394,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 201334,\n'
                                             '        "iops_max" : 202338,\n'
                                             '        "iops_mean" : '
                                             '201700.000000,\n'
                                             '        "iops_stddev" : '
                                             '285.374787,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 21.028571,\n'
                                             '      "sys_cpu" : 47.785714,\n'
                                             '      "ctx" : 705798,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 91.422546,\n'
                                             '        "20" : 8.482674,\n'
                                             '        "50" : 0.060565,\n'
                                             '        "100" : 0.022597,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=140286: Wed '
                                             'Oct 26 04:31:06 2022\n'
                                             '  read: IOPS=202k, BW=98.5MiB/s '
                                             '(103MB/s)(689MiB/7001msec)\n'
                                             '    slat (nsec): min=1202, '
                                             'max=25348, avg=1368.42, '
                                             'stdev=146.45\n'
                                             '    clat (nsec): min=120, '
                                             'max=109946, avg=8258.92, '
                                             'stdev=2093.68\n'
                                             '     lat (usec): min=6, max=111, '
                                             'avg= 9.63, stdev= 2.16\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5472],  '
                                             '5.00th=[ 6240], 10.00th=[ 6240], '
                                             '20.00th=[ 6304],\n'
                                             '     | 30.00th=[ 6432], '
                                             '40.00th=[ 7776], 50.00th=[ '
                                             '8032], 60.00th=[ 8256],\n'
                                             '     | 70.00th=[ 9792], '
                                             '80.00th=[ 9920], 90.00th=[ '
                                             '9920], 95.00th=[10048],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[17024], '
                                             '99.90th=[18560], '
                                             '99.95th=[23680],\n'
                                             '     | 99.99th=[60672]\n'
                                             '   bw (  KiB/s): min=100667, '
                                             'max=101169, per=100.00%, '
                                             'avg=100850.00, stdev=142.69, '
                                             'samples=14\n'
                                             '   iops        : min=201334, '
                                             'max=202338, avg=201700.00, '
                                             'stdev=285.37, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=91.42%, 20=8.48%, '
                                             '50=0.06%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=21.03%, '
                                             'sys=47.79%, ctx=705798, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1411701,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=98.5MiB/s (103MB/s), '
                                             '98.5MiB/s-98.5MiB/s '
                                             '(103MB/s-103MB/s), io=689MiB '
                                             '(723MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.output'),
                                   'state': {'begin': 1666744377.02161,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7968065738677979,
                                             'end': 1666744377.8184166,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_114.state')},
                       'cmd_115': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.output'),
                                   'state': {'begin': 1666744378.8055432,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7906351089477539,
                                             'end': 1666744379.5961783,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_115.state')},
                       'cmd_116': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.output'),
                                   'state': {'begin': 1666744379.598312,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.064814329147339,
                                             'end': 1666744390.6631262,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_116.state')},
                       'cmd_117': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751479,\n'
                                             '  "timestamp_ms" : '
                                             '1666751479857,\n'
                                             '  "time" : "Wed Oct 26 04:31:19 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 727922688,\n'
                                             '        "io_kbytes" : 710862,\n'
                                             '        "bw_bytes" : 103974101,\n'
                                             '        "bw" : 101537,\n'
                                             '        "iops" : 203074.275104,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1421723,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1212,\n'
                                             '          "max" : 24436,\n'
                                             '          "mean" : 1384.957250,\n'
                                             '          "stddev" : '
                                             '152.943213,\n'
                                             '          "N" : 1421723\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 120,\n'
                                             '          "max" : 77255,\n'
                                             '          "mean" : 8203.277642,\n'
                                             '          "stddev" : '
                                             '2113.140529,\n'
                                             '          "N" : 1421724,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5280,\n'
                                             '            "5.000000" : 6176,\n'
                                             '            "10.000000" : 6240,\n'
                                             '            "20.000000" : 6304,\n'
                                             '            "30.000000" : 6432,\n'
                                             '            "40.000000" : 7776,\n'
                                             '            "50.000000" : 8032,\n'
                                             '            "60.000000" : 8256,\n'
                                             '            "70.000000" : 9792,\n'
                                             '            "80.000000" : 9920,\n'
                                             '            "90.000000" : 9920,\n'
                                             '            "95.000000" : '
                                             '10048,\n'
                                             '            "99.000000" : '
                                             '16320,\n'
                                             '            "99.500000" : '
                                             '17280,\n'
                                             '            "99.900000" : '
                                             '18816,\n'
                                             '            "99.950000" : '
                                             '23424,\n'
                                             '            "99.990000" : 60672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6181,\n'
                                             '          "max" : 78517,\n'
                                             '          "mean" : 9588.234953,\n'
                                             '          "stddev" : '
                                             '2180.631735,\n'
                                             '          "N" : 1421724\n'
                                             '        },\n'
                                             '        "bw_min" : 101351,\n'
                                             '        "bw_max" : 101836,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '101561.714286,\n'
                                             '        "bw_dev" : 140.942666,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 202702,\n'
                                             '        "iops_max" : 203672,\n'
                                             '        "iops_mean" : '
                                             '203123.428571,\n'
                                             '        "iops_stddev" : '
                                             '281.885332,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.828571,\n'
                                             '      "sys_cpu" : 49.057143,\n'
                                             '      "ctx" : 710806,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 92.513310,\n'
                                             '        "20" : 7.385335,\n'
                                             '        "50" : 0.065695,\n'
                                             '        "100" : 0.023774,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=140325: Wed '
                                             'Oct 26 04:31:19 2022\n'
                                             '  read: IOPS=203k, BW=99.2MiB/s '
                                             '(104MB/s)(694MiB/7001msec)\n'
                                             '    slat (nsec): min=1212, '
                                             'max=24436, avg=1384.96, '
                                             'stdev=152.94\n'
                                             '    clat (nsec): min=120, '
                                             'max=77255, avg=8203.28, '
                                             'stdev=2113.14\n'
                                             '     lat (nsec): min=6181, '
                                             'max=78517, avg=9588.23, '
                                             'stdev=2180.63\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5280],  '
                                             '5.00th=[ 6176], 10.00th=[ 6240], '
                                             '20.00th=[ 6304],\n'
                                             '     | 30.00th=[ 6432], '
                                             '40.00th=[ 7776], 50.00th=[ '
                                             '8032], 60.00th=[ 8256],\n'
                                             '     | 70.00th=[ 9792], '
                                             '80.00th=[ 9920], 90.00th=[ '
                                             '9920], 95.00th=[10048],\n'
                                             '     | 99.00th=[16320], '
                                             '99.50th=[17280], '
                                             '99.90th=[18816], '
                                             '99.95th=[23424],\n'
                                             '     | 99.99th=[60672]\n'
                                             '   bw (  KiB/s): min=101351, '
                                             'max=101836, per=100.00%, '
                                             'avg=101561.71, stdev=140.94, '
                                             'samples=14\n'
                                             '   iops        : min=202702, '
                                             'max=203672, avg=203123.43, '
                                             'stdev=281.89, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=92.51%, 20=7.39%, '
                                             '50=0.07%\n'
                                             '  lat (usec)   : 100=0.02%\n'
                                             '  cpu          : usr=19.83%, '
                                             'sys=49.06%, ctx=710806, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1421723,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=99.2MiB/s (104MB/s), '
                                             '99.2MiB/s-99.2MiB/s '
                                             '(104MB/s-104MB/s), io=694MiB '
                                             '(728MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.output'),
                                   'state': {'begin': 1666744390.6660898,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7893517017364502,
                                             'end': 1666744391.4554415,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_117.state')},
                       'cmd_118': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.output'),
                                   'state': {'begin': 1666744393.4771495,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.782909631729126,
                                             'end': 1666744394.260059,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_118.state')},
                       'cmd_119': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.output'),
                                   'state': {'begin': 1666744394.2621856,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.489806890487671,
                                             'end': 1666744405.7519925,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_119.state')},
                       'cmd_12': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666750981,\n'
                                            '  "timestamp_ms" : '
                                            '1666750981040,\n'
                                            '  "time" : "Wed Oct 26 04:23:01 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring_cmd",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 290464256,\n'
                                            '        "io_kbytes" : 283656,\n'
                                            '        "bw_bytes" : 41488966,\n'
                                            '        "bw" : 40516,\n'
                                            '        "iops" : 81032.995286,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 567312,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2304,\n'
                                            '          "max" : 40236,\n'
                                            '          "mean" : 2576.996690,\n'
                                            '          "stddev" : 232.578518,\n'
                                            '          "N" : 567313\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 340,\n'
                                            '          "max" : 70823,\n'
                                            '          "mean" : 9183.791792,\n'
                                            '          "stddev" : '
                                            '1245.876786,\n'
                                            '          "N" : 567312,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8768,\n'
                                            '            "5.000000" : 8896,\n'
                                            '            "10.000000" : 8896,\n'
                                            '            "20.000000" : 8896,\n'
                                            '            "30.000000" : 8896,\n'
                                            '            "40.000000" : 9024,\n'
                                            '            "50.000000" : 9024,\n'
                                            '            "60.000000" : 9024,\n'
                                            '            "70.000000" : 9024,\n'
                                            '            "80.000000" : 9024,\n'
                                            '            "90.000000" : 9152,\n'
                                            '            "95.000000" : 9280,\n'
                                            '            "99.000000" : 15552,\n'
                                            '            "99.500000" : 16512,\n'
                                            '            "99.900000" : 19328,\n'
                                            '            "99.950000" : 21632,\n'
                                            '            "99.990000" : 55552\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7985,\n'
                                            '          "max" : 73398,\n'
                                            '          "mean" : 11760.788467,\n'
                                            '          "stddev" : '
                                            '1263.053401,\n'
                                            '          "N" : 567312\n'
                                            '        },\n'
                                            '        "bw_min" : 40426,\n'
                                            '        "bw_max" : 40799,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '40525.857143,\n'
                                            '        "bw_dev" : 100.937038,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 80852,\n'
                                            '        "iops_max" : 81598,\n'
                                            '        "iops_mean" : '
                                            '81051.714286,\n'
                                            '        "iops_stddev" : '
                                            '201.874077,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 14.071429,\n'
                                            '      "sys_cpu" : 45.314286,\n'
                                            '      "ctx" : 567166,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.013220,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 97.317525,\n'
                                            '        "20" : 2.597512,\n'
                                            '        "50" : 0.056583,\n'
                                            '        "100" : 0.014983,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=138790: Wed '
                                            'Oct 26 04:23:01 2022\n'
                                            '  read: IOPS=81.0k, BW=39.6MiB/s '
                                            '(41.5MB/s)(277MiB/7001msec)\n'
                                            '    slat (nsec): min=2304, '
                                            'max=40236, avg=2577.00, '
                                            'stdev=232.58\n'
                                            '    clat (nsec): min=340, '
                                            'max=70823, avg=9183.79, '
                                            'stdev=1245.88\n'
                                            '     lat (nsec): min=7985, '
                                            'max=73398, avg=11760.79, '
                                            'stdev=1263.05\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8768],  5.00th=[ '
                                            '8896], 10.00th=[ 8896], 20.00th=[ '
                                            '8896],\n'
                                            '     | 30.00th=[ 8896], 40.00th=[ '
                                            '9024], 50.00th=[ 9024], 60.00th=[ '
                                            '9024],\n'
                                            '     | 70.00th=[ 9024], 80.00th=[ '
                                            '9024], 90.00th=[ 9152], 95.00th=[ '
                                            '9280],\n'
                                            '     | 99.00th=[15552], '
                                            '99.50th=[16512], 99.90th=[19328], '
                                            '99.95th=[21632],\n'
                                            '     | 99.99th=[55552]\n'
                                            '   bw (  KiB/s): min=40426, '
                                            'max=40799, per=100.00%, '
                                            'avg=40525.86, stdev=100.94, '
                                            'samples=14\n'
                                            '   iops        : min=80852, '
                                            'max=81598, avg=81051.71, '
                                            'stdev=201.87, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%, '
                                            '750=0.01%\n'
                                            '  lat (usec)   : 10=97.32%, '
                                            '20=2.60%, 50=0.06%, 100=0.01%\n'
                                            '  cpu          : usr=14.07%, '
                                            'sys=45.31%, ctx=567166, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=567312,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=39.6MiB/s (41.5MB/s), '
                                            '39.6MiB/s-39.6MiB/s '
                                            '(41.5MB/s-41.5MB/s), io=277MiB '
                                            '(290MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.output'),
                                  'state': {'begin': 1666743891.8404493,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7901482582092285,
                                            'end': 1666743892.6305976,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_12.state')},
                       'cmd_120': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751494,\n'
                                             '  "timestamp_ms" : '
                                             '1666751494875,\n'
                                             '  "time" : "Wed Oct 26 04:31:34 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 840442368,\n'
                                             '        "io_kbytes" : 820744,\n'
                                             '        "bw_bytes" : 120046045,\n'
                                             '        "bw" : 117232,\n'
                                             '        "iops" : 234464.790744,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1641488,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 230,\n'
                                             '          "max" : 20989,\n'
                                             '          "mean" : 273.106698,\n'
                                             '          "stddev" : 42.767660,\n'
                                             '          "N" : 1641488\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 270,\n'
                                             '          "max" : 112260,\n'
                                             '          "mean" : 8033.977383,\n'
                                             '          "stddev" : '
                                             '2024.239647,\n'
                                             '          "N" : 1641489,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6368,\n'
                                             '            "5.000000" : 6432,\n'
                                             '            "10.000000" : 6496,\n'
                                             '            "20.000000" : 6944,\n'
                                             '            "30.000000" : 7392,\n'
                                             '            "40.000000" : 7456,\n'
                                             '            "50.000000" : 7648,\n'
                                             '            "60.000000" : 7840,\n'
                                             '            "70.000000" : 8640,\n'
                                             '            "80.000000" : 8768,\n'
                                             '            "90.000000" : 8896,\n'
                                             '            "95.000000" : 9024,\n'
                                             '            "99.000000" : '
                                             '18816,\n'
                                             '            "99.500000" : '
                                             '20096,\n'
                                             '            "99.900000" : '
                                             '22400,\n'
                                             '            "99.950000" : '
                                             '27776,\n'
                                             '            "99.990000" : 62208\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6081,\n'
                                             '          "max" : 112541,\n'
                                             '          "mean" : 8307.084073,\n'
                                             '          "stddev" : '
                                             '2025.511582,\n'
                                             '          "N" : 1641489\n'
                                             '        },\n'
                                             '        "bw_min" : 116935,\n'
                                             '        "bw_max" : 117650,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '117259.714286,\n'
                                             '        "bw_dev" : 195.708350,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 233870,\n'
                                             '        "iops_max" : 235300,\n'
                                             '        "iops_mean" : '
                                             '234519.428571,\n'
                                             '        "iops_stddev" : '
                                             '391.416700,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 25,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.981480,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.879478,\n'
                                             '        "20" : 2.601055,\n'
                                             '        "50" : 0.497293,\n'
                                             '        "100" : 0.021505,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=140374: Wed Oct 26 04:31:34 '
                                             '2022\n'
                                             '  read: IOPS=234k, BW=114MiB/s '
                                             '(120MB/s)(802MiB/7001msec)\n'
                                             '    slat (nsec): min=230, '
                                             'max=20989, avg=273.11, '
                                             'stdev=42.77\n'
                                             '    clat (nsec): min=270, '
                                             'max=112260, avg=8033.98, '
                                             'stdev=2024.24\n'
                                             '     lat (usec): min=6, max=112, '
                                             'avg= 8.31, stdev= 2.03\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6368],  '
                                             '5.00th=[ 6432], 10.00th=[ 6496], '
                                             '20.00th=[ 6944],\n'
                                             '     | 30.00th=[ 7392], '
                                             '40.00th=[ 7456], 50.00th=[ '
                                             '7648], 60.00th=[ 7840],\n'
                                             '     | 70.00th=[ 8640], '
                                             '80.00th=[ 8768], 90.00th=[ '
                                             '8896], 95.00th=[ 9024],\n'
                                             '     | 99.00th=[18816], '
                                             '99.50th=[20096], '
                                             '99.90th=[22400], '
                                             '99.95th=[27776],\n'
                                             '     | 99.99th=[62208]\n'
                                             '   bw (  KiB/s): min=116935, '
                                             'max=117650, per=100.00%, '
                                             'avg=117259.71, stdev=195.71, '
                                             'samples=14\n'
                                             '   iops        : min=233870, '
                                             'max=235300, avg=234519.43, '
                                             'stdev=391.42, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%, '
                                             '750=0.01%, 1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=96.88%, 20=2.60%, '
                                             '50=0.50%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=25, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1641488,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=114MiB/s (120MB/s), '
                                             '114MiB/s-114MiB/s '
                                             '(120MB/s-120MB/s), io=802MiB '
                                             '(840MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.output'),
                                   'state': {'begin': 1666744405.7548473,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.794299840927124,
                                             'end': 1666744406.5491471,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_120.state')},
                       'cmd_121': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.output'),
                                   'state': {'begin': 1666744408.5745897,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7830724716186523,
                                             'end': 1666744409.3576622,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_121.state')},
                       'cmd_122': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.output'),
                                   'state': {'begin': 1666744409.3596444,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.487300872802734,
                                             'end': 1666744420.8469453,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_122.state')},
                       'cmd_123': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751509,\n'
                                             '  "timestamp_ms" : '
                                             '1666751509970,\n'
                                             '  "time" : "Wed Oct 26 04:31:49 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 906781696,\n'
                                             '        "io_kbytes" : 885529,\n'
                                             '        "bw_bytes" : 129521739,\n'
                                             '        "bw" : 126486,\n'
                                             '        "iops" : 252972.003999,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1771057,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 210,\n'
                                             '          "max" : 10360,\n'
                                             '          "mean" : 266.127212,\n'
                                             '          "stddev" : 24.148686,\n'
                                             '          "N" : 1771057\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 281,\n'
                                             '          "max" : 77735,\n'
                                             '          "mean" : 7425.552159,\n'
                                             '          "stddev" : '
                                             '1737.669617,\n'
                                             '          "N" : 1771058,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6816,\n'
                                             '            "5.000000" : 7008,\n'
                                             '            "10.000000" : 7008,\n'
                                             '            "20.000000" : 7072,\n'
                                             '            "30.000000" : 7072,\n'
                                             '            "40.000000" : 7136,\n'
                                             '            "50.000000" : 7200,\n'
                                             '            "60.000000" : 7200,\n'
                                             '            "70.000000" : 7264,\n'
                                             '            "80.000000" : 7328,\n'
                                             '            "90.000000" : 7392,\n'
                                             '            "95.000000" : 7520,\n'
                                             '            "99.000000" : '
                                             '16320,\n'
                                             '            "99.500000" : '
                                             '20864,\n'
                                             '            "99.900000" : '
                                             '22144,\n'
                                             '            "99.950000" : '
                                             '22912,\n'
                                             '            "99.990000" : 66048\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5530,\n'
                                             '          "max" : 77996,\n'
                                             '          "mean" : 7691.679367,\n'
                                             '          "stddev" : '
                                             '1738.017364,\n'
                                             '          "N" : 1771058\n'
                                             '        },\n'
                                             '        "bw_min" : 126151,\n'
                                             '        "bw_max" : 126861,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '126549.785714,\n'
                                             '        "bw_dev" : 246.186039,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 252302,\n'
                                             '        "iops_max" : 253722,\n'
                                             '        "iops_mean" : '
                                             '253099.642857,\n'
                                             '        "iops_stddev" : '
                                             '492.334097,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 23,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.990401,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.000831,\n'
                                             '        "20" : 2.473890,\n'
                                             '        "50" : 0.499589,\n'
                                             '        "100" : 0.025070,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=140424: Wed Oct 26 04:31:49 '
                                             '2022\n'
                                             '  read: IOPS=253k, BW=124MiB/s '
                                             '(130MB/s)(865MiB/7001msec)\n'
                                             '    slat (nsec): min=210, '
                                             'max=10360, avg=266.13, '
                                             'stdev=24.15\n'
                                             '    clat (nsec): min=281, '
                                             'max=77735, avg=7425.55, '
                                             'stdev=1737.67\n'
                                             '     lat (nsec): min=5530, '
                                             'max=77996, avg=7691.68, '
                                             'stdev=1738.02\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6816],  '
                                             '5.00th=[ 7008], 10.00th=[ 7008], '
                                             '20.00th=[ 7072],\n'
                                             '     | 30.00th=[ 7072], '
                                             '40.00th=[ 7136], 50.00th=[ '
                                             '7200], 60.00th=[ 7200],\n'
                                             '     | 70.00th=[ 7264], '
                                             '80.00th=[ 7328], 90.00th=[ '
                                             '7392], 95.00th=[ 7520],\n'
                                             '     | 99.00th=[16320], '
                                             '99.50th=[20864], '
                                             '99.90th=[22144], '
                                             '99.95th=[22912],\n'
                                             '     | 99.99th=[66048]\n'
                                             '   bw (  KiB/s): min=126151, '
                                             'max=126861, per=100.00%, '
                                             'avg=126549.79, stdev=246.19, '
                                             'samples=14\n'
                                             '   iops        : min=252302, '
                                             'max=253722, avg=253099.64, '
                                             'stdev=492.33, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.00%, 20=2.47%, 50=0.50%, '
                                             '100=0.03%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=23, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1771057,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=124MiB/s (130MB/s), '
                                             '124MiB/s-124MiB/s '
                                             '(130MB/s-130MB/s), io=865MiB '
                                             '(907MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.output'),
                                   'state': {'begin': 1666744420.8497894,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7872164249420166,
                                             'end': 1666744421.6370058,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_123.state')},
                       'cmd_124': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.output'),
                                   'state': {'begin': 1666744423.6496933,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7778506278991699,
                                             'end': 1666744424.4275439,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_124.state')},
                       'cmd_125': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.output'),
                                   'state': {'begin': 1666744424.4295397,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.47445011138916,
                                             'end': 1666744435.9039898,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_125.state')},
                       'cmd_126': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751525,\n'
                                             '  "timestamp_ms" : '
                                             '1666751525027,\n'
                                             '  "time" : "Wed Oct 26 04:32:05 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 834501120,\n'
                                             '        "io_kbytes" : 814942,\n'
                                             '        "bw_bytes" : 119197417,\n'
                                             '        "bw" : 116403,\n'
                                             '        "iops" : 232807.313241,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1629884,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 230,\n'
                                             '          "max" : 27211,\n'
                                             '          "mean" : 278.884964,\n'
                                             '          "stddev" : 29.538508,\n'
                                             '          "N" : 1629884\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 872,\n'
                                             '          "max" : 121678,\n'
                                             '          "mean" : 8094.376448,\n'
                                             '          "stddev" : '
                                             '2042.240059,\n'
                                             '          "N" : 1629885,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6368,\n'
                                             '            "5.000000" : 6432,\n'
                                             '            "10.000000" : 6560,\n'
                                             '            "20.000000" : 7264,\n'
                                             '            "30.000000" : 7392,\n'
                                             '            "40.000000" : 7520,\n'
                                             '            "50.000000" : 7648,\n'
                                             '            "60.000000" : 7904,\n'
                                             '            "70.000000" : 8768,\n'
                                             '            "80.000000" : 8896,\n'
                                             '            "90.000000" : 8896,\n'
                                             '            "95.000000" : 9024,\n'
                                             '            "99.000000" : '
                                             '18816,\n'
                                             '            "99.500000" : '
                                             '20096,\n'
                                             '            "99.900000" : '
                                             '22400,\n'
                                             '            "99.950000" : '
                                             '28544,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5881,\n'
                                             '          "max" : 121959,\n'
                                             '          "mean" : 8373.261407,\n'
                                             '          "stddev" : '
                                             '2043.377005,\n'
                                             '          "N" : 1629885\n'
                                             '        },\n'
                                             '        "bw_min" : 116102,\n'
                                             '        "bw_max" : 116981,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '116429.500000,\n'
                                             '        "bw_dev" : 245.284372,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 232204,\n'
                                             '        "iops_max" : 233962,\n'
                                             '        "iops_mean" : '
                                             '232859.357143,\n'
                                             '        "iops_stddev" : '
                                             '490.490744,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 23,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.990551,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 96.857322,\n'
                                             '        "20" : 2.646017,\n'
                                             '        "50" : 0.471567,\n'
                                             '        "100" : 0.024726,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=140474: Wed Oct 26 04:32:05 '
                                             '2022\n'
                                             '  read: IOPS=233k, BW=114MiB/s '
                                             '(119MB/s)(796MiB/7001msec)\n'
                                             '    slat (nsec): min=230, '
                                             'max=27211, avg=278.88, '
                                             'stdev=29.54\n'
                                             '    clat (nsec): min=872, '
                                             'max=121678, avg=8094.38, '
                                             'stdev=2042.24\n'
                                             '     lat (usec): min=5, max=121, '
                                             'avg= 8.37, stdev= 2.04\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6368],  '
                                             '5.00th=[ 6432], 10.00th=[ 6560], '
                                             '20.00th=[ 7264],\n'
                                             '     | 30.00th=[ 7392], '
                                             '40.00th=[ 7520], 50.00th=[ '
                                             '7648], 60.00th=[ 7904],\n'
                                             '     | 70.00th=[ 8768], '
                                             '80.00th=[ 8896], 90.00th=[ '
                                             '8896], 95.00th=[ 9024],\n'
                                             '     | 99.00th=[18816], '
                                             '99.50th=[20096], '
                                             '99.90th=[22400], '
                                             '99.95th=[28544],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=116102, '
                                             'max=116981, per=100.00%, '
                                             'avg=116429.50, stdev=245.28, '
                                             'samples=14\n'
                                             '   iops        : min=232204, '
                                             'max=233962, avg=232859.36, '
                                             'stdev=490.49, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 10=96.86%, '
                                             '20=2.65%, 50=0.47%, 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=23, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1629884,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=114MiB/s (119MB/s), '
                                             '114MiB/s-114MiB/s '
                                             '(119MB/s-119MB/s), io=796MiB '
                                             '(835MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.output'),
                                   'state': {'begin': 1666744435.9070897,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7856025695800781,
                                             'end': 1666744436.6926923,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_126.state')},
                       'cmd_127': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.output'),
                                   'state': {'begin': 1666744437.6836796,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7869439125061035,
                                             'end': 1666744438.4706235,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_127.state')},
                       'cmd_128': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.output'),
                                   'state': {'begin': 1666744438.4730957,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.045161724090576,
                                             'end': 1666744449.5182574,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_128.state')},
                       'cmd_129': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751538,\n'
                                             '  "timestamp_ms" : '
                                             '1666751538710,\n'
                                             '  "time" : "Wed Oct 26 04:32:18 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 712784384,\n'
                                             '        "io_kbytes" : 696078,\n'
                                             '        "bw_bytes" : 101811796,\n'
                                             '        "bw" : 99425,\n'
                                             '        "iops" : 198851.021283,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1392156,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1282,\n'
                                             '          "max" : 23945,\n'
                                             '          "mean" : 1414.855360,\n'
                                             '          "stddev" : '
                                             '133.152742,\n'
                                             '          "N" : 1392156\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 80,\n'
                                             '          "max" : 100759,\n'
                                             '          "mean" : 8354.827444,\n'
                                             '          "stddev" : '
                                             '2236.779986,\n'
                                             '          "N" : 1392157,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5536,\n'
                                             '            "5.000000" : 6240,\n'
                                             '            "10.000000" : 6304,\n'
                                             '            "20.000000" : 6368,\n'
                                             '            "30.000000" : 6432,\n'
                                             '            "40.000000" : 6496,\n'
                                             '            "50.000000" : 8256,\n'
                                             '            "60.000000" : 9792,\n'
                                             '            "70.000000" : 9920,\n'
                                             '            "80.000000" : '
                                             '10048,\n'
                                             '            "90.000000" : '
                                             '10048,\n'
                                             '            "95.000000" : '
                                             '10176,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '17024,\n'
                                             '            "99.900000" : '
                                             '19072,\n'
                                             '            "99.950000" : '
                                             '23168,\n'
                                             '            "99.990000" : 60160\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6342,\n'
                                             '          "max" : 102081,\n'
                                             '          "mean" : 9769.682853,\n'
                                             '          "stddev" : '
                                             '2306.249536,\n'
                                             '          "N" : 1392157\n'
                                             '        },\n'
                                             '        "bw_min" : 99158,\n'
                                             '        "bw_max" : 99923,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '99467.928571,\n'
                                             '        "bw_dev" : 285.062062,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 198316,\n'
                                             '        "iops_max" : 199846,\n'
                                             '        "iops_mean" : '
                                             '198936.071429,\n'
                                             '        "iops_stddev" : '
                                             '570.072932,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 18.328571,\n'
                                             '      "sys_cpu" : 51.842857,\n'
                                             '      "ctx" : 696330,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 81.288448,\n'
                                             '        "20" : 18.611635,\n'
                                             '        "50" : 0.069820,\n'
                                             '        "100" : 0.020687,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2002080,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 10829,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 10829,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=140513: Wed '
                                             'Oct 26 04:32:18 2022\n'
                                             '  read: IOPS=199k, BW=97.1MiB/s '
                                             '(102MB/s)(680MiB/7001msec)\n'
                                             '    slat (nsec): min=1282, '
                                             'max=23945, avg=1414.86, '
                                             'stdev=133.15\n'
                                             '    clat (nsec): min=80, '
                                             'max=100759, avg=8354.83, '
                                             'stdev=2236.78\n'
                                             '     lat (usec): min=6, max=102, '
                                             'avg= 9.77, stdev= 2.31\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5536],  '
                                             '5.00th=[ 6240], 10.00th=[ 6304], '
                                             '20.00th=[ 6368],\n'
                                             '     | 30.00th=[ 6432], '
                                             '40.00th=[ 6496], 50.00th=[ '
                                             '8256], 60.00th=[ 9792],\n'
                                             '     | 70.00th=[ 9920], '
                                             '80.00th=[10048], '
                                             '90.00th=[10048], '
                                             '95.00th=[10176],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[17024], '
                                             '99.90th=[19072], '
                                             '99.95th=[23168],\n'
                                             '     | 99.99th=[60160]\n'
                                             '   bw (  KiB/s): min=99158, '
                                             'max=99923, per=100.00%, '
                                             'avg=99467.93, stdev=285.06, '
                                             'samples=14\n'
                                             '   iops        : min=198316, '
                                             'max=199846, avg=198936.07, '
                                             'stdev=570.07, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=81.29%, 20=18.61%, '
                                             '50=0.07%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=18.33%, '
                                             'sys=51.84%, ctx=696330, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1392156,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=97.1MiB/s (102MB/s), '
                                             '97.1MiB/s-97.1MiB/s '
                                             '(102MB/s-102MB/s), io=680MiB '
                                             '(713MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2002080/0, '
                                             'merge=0/0, ticks=10829/0, '
                                             'in_queue=10829, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.output'),
                                   'state': {'begin': 1666744449.5212543,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8464257717132568,
                                             'end': 1666744450.36768,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_129.state')},
                       'cmd_13': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.output'),
                                  'state': {'begin': 1666743893.626948,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7838962078094482,
                                            'end': 1666743894.4108443,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_13.state')},
                       'cmd_130': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.output'),
                                   'state': {'begin': 1666744451.4471276,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.922795295715332,
                                             'end': 1666744452.3699229,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_130.state')},
                       'cmd_131': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.output'),
                                   'state': {'begin': 1666744452.3719704,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.149224996566772,
                                             'end': 1666744463.5211954,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_131.state')},
                       'cmd_132': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751552,\n'
                                             '  "timestamp_ms" : '
                                             '1666751552714,\n'
                                             '  "time" : "Wed Oct 26 04:32:32 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 713009152,\n'
                                             '        "io_kbytes" : 696298,\n'
                                             '        "bw_bytes" : 101843901,\n'
                                             '        "bw" : 99456,\n'
                                             '        "iops" : 198913.726610,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1392595,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1282,\n'
                                             '          "max" : 49703,\n'
                                             '          "mean" : 1410.760628,\n'
                                             '          "stddev" : '
                                             '136.155381,\n'
                                             '          "N" : 1392595\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 80,\n'
                                             '          "max" : 102632,\n'
                                             '          "mean" : 8358.082344,\n'
                                             '          "stddev" : '
                                             '2240.215117,\n'
                                             '          "N" : 1392596,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5536,\n'
                                             '            "5.000000" : 6304,\n'
                                             '            "10.000000" : 6304,\n'
                                             '            "20.000000" : 6368,\n'
                                             '            "30.000000" : 6432,\n'
                                             '            "40.000000" : 6496,\n'
                                             '            "50.000000" : 8256,\n'
                                             '            "60.000000" : 9792,\n'
                                             '            "70.000000" : 9920,\n'
                                             '            "80.000000" : 9920,\n'
                                             '            "90.000000" : '
                                             '10048,\n'
                                             '            "95.000000" : '
                                             '10176,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '17024,\n'
                                             '            "99.900000" : '
                                             '19072,\n'
                                             '            "99.950000" : '
                                             '23168,\n'
                                             '            "99.990000" : 65280\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5671,\n'
                                             '          "max" : 103965,\n'
                                             '          "mean" : 9768.843017,\n'
                                             '          "stddev" : '
                                             '2302.931642,\n'
                                             '          "N" : 1392596\n'
                                             '        },\n'
                                             '        "bw_min" : 99307,\n'
                                             '        "bw_max" : 99675,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '99497.357143,\n'
                                             '        "bw_dev" : 102.195595,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 198614,\n'
                                             '        "iops_max" : 199350,\n'
                                             '        "iops_mean" : '
                                             '198994.785714,\n'
                                             '        "iops_stddev" : '
                                             '204.325599,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.071429,\n'
                                             '      "sys_cpu" : 51.157143,\n'
                                             '      "ctx" : 696426,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 83.180681,\n'
                                             '        "20" : 16.725394,\n'
                                             '        "50" : 0.061612,\n'
                                             '        "100" : 0.023625,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2000153,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 10851,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 10851,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=140551: Wed '
                                             'Oct 26 04:32:32 2022\n'
                                             '  read: IOPS=199k, BW=97.1MiB/s '
                                             '(102MB/s)(680MiB/7001msec)\n'
                                             '    slat (nsec): min=1282, '
                                             'max=49703, avg=1410.76, '
                                             'stdev=136.16\n'
                                             '    clat (nsec): min=80, '
                                             'max=102632, avg=8358.08, '
                                             'stdev=2240.22\n'
                                             '     lat (usec): min=5, max=103, '
                                             'avg= 9.77, stdev= 2.30\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5536],  '
                                             '5.00th=[ 6304], 10.00th=[ 6304], '
                                             '20.00th=[ 6368],\n'
                                             '     | 30.00th=[ 6432], '
                                             '40.00th=[ 6496], 50.00th=[ '
                                             '8256], 60.00th=[ 9792],\n'
                                             '     | 70.00th=[ 9920], '
                                             '80.00th=[ 9920], '
                                             '90.00th=[10048], '
                                             '95.00th=[10176],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[17024], '
                                             '99.90th=[19072], '
                                             '99.95th=[23168],\n'
                                             '     | 99.99th=[65280]\n'
                                             '   bw (  KiB/s): min=99307, '
                                             'max=99675, per=100.00%, '
                                             'avg=99497.36, stdev=102.20, '
                                             'samples=14\n'
                                             '   iops        : min=198614, '
                                             'max=199350, avg=198994.79, '
                                             'stdev=204.33, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=83.18%, 20=16.73%, '
                                             '50=0.06%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=19.07%, '
                                             'sys=51.16%, ctx=696426, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1392595,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=97.1MiB/s (102MB/s), '
                                             '97.1MiB/s-97.1MiB/s '
                                             '(102MB/s-102MB/s), io=680MiB '
                                             '(713MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2000153/0, '
                                             'merge=0/0, ticks=10851/0, '
                                             'in_queue=10851, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.output'),
                                   'state': {'begin': 1666744463.523983,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.0828144550323486,
                                             'end': 1666744464.6067975,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_132.state')},
                       'cmd_133': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.output'),
                                   'state': {'begin': 1666744465.8849888,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 1.1885249614715576,
                                             'end': 1666744467.0735137,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_133.state')},
                       'cmd_134': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.output'),
                                   'state': {'begin': 1666744467.0755863,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.199074029922485,
                                             'end': 1666744478.2746603,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_134.state')},
                       'cmd_135': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751567,\n'
                                             '  "timestamp_ms" : '
                                             '1666751567468,\n'
                                             '  "time" : "Wed Oct 26 04:32:47 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 704743424,\n'
                                             '        "io_kbytes" : 688226,\n'
                                             '        "bw_bytes" : 100663251,\n'
                                             '        "bw" : 98303,\n'
                                             '        "iops" : 196607.770319,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1376451,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1302,\n'
                                             '          "max" : 115346,\n'
                                             '          "mean" : 1448.851801,\n'
                                             '          "stddev" : '
                                             '170.761925,\n'
                                             '          "N" : 1376451\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 90,\n'
                                             '          "max" : 124113,\n'
                                             '          "mean" : 8432.095208,\n'
                                             '          "stddev" : '
                                             '2364.959566,\n'
                                             '          "N" : 1376452,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5344,\n'
                                             '            "5.000000" : 6304,\n'
                                             '            "10.000000" : 6304,\n'
                                             '            "20.000000" : 6368,\n'
                                             '            "30.000000" : 6432,\n'
                                             '            "40.000000" : 6496,\n'
                                             '            "50.000000" : 8896,\n'
                                             '            "60.000000" : '
                                             '10048,\n'
                                             '            "70.000000" : '
                                             '10048,\n'
                                             '            "80.000000" : '
                                             '10176,\n'
                                             '            "90.000000" : '
                                             '10176,\n'
                                             '            "95.000000" : '
                                             '10304,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '17280,\n'
                                             '            "99.900000" : '
                                             '23168,\n'
                                             '            "99.950000" : '
                                             '23424,\n'
                                             '            "99.990000" : 65280\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6342,\n'
                                             '          "max" : 125696,\n'
                                             '          "mean" : 9880.947041,\n'
                                             '          "stddev" : '
                                             '2442.322795,\n'
                                             '          "N" : 1376452\n'
                                             '        },\n'
                                             '        "bw_min" : 98131,\n'
                                             '        "bw_max" : 98526,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '98347.384615,\n'
                                             '        "bw_dev" : 105.430497,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 196262,\n'
                                             '        "iops_max" : 197052,\n'
                                             '        "iops_mean" : '
                                             '196694.846154,\n'
                                             '        "iops_stddev" : '
                                             '210.890116,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 18.671429,\n'
                                             '      "sys_cpu" : 52.142857,\n'
                                             '      "ctx" : 689064,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 59.597109,\n'
                                             '        "20" : 40.243205,\n'
                                             '        "50" : 0.122925,\n'
                                             '        "100" : 0.025064,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 1979948,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 10785,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 10785,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=140589: Wed '
                                             'Oct 26 04:32:47 2022\n'
                                             '  read: IOPS=197k, BW=96.0MiB/s '
                                             '(101MB/s)(672MiB/7001msec)\n'
                                             '    slat (nsec): min=1302, '
                                             'max=115346, avg=1448.85, '
                                             'stdev=170.76\n'
                                             '    clat (nsec): min=90, '
                                             'max=124113, avg=8432.10, '
                                             'stdev=2364.96\n'
                                             '     lat (usec): min=6, max=125, '
                                             'avg= 9.88, stdev= 2.44\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5344],  '
                                             '5.00th=[ 6304], 10.00th=[ 6304], '
                                             '20.00th=[ 6368],\n'
                                             '     | 30.00th=[ 6432], '
                                             '40.00th=[ 6496], 50.00th=[ '
                                             '8896], 60.00th=[10048],\n'
                                             '     | 70.00th=[10048], '
                                             '80.00th=[10176], '
                                             '90.00th=[10176], '
                                             '95.00th=[10304],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[17280], '
                                             '99.90th=[23168], '
                                             '99.95th=[23424],\n'
                                             '     | 99.99th=[65280]\n'
                                             '   bw (  KiB/s): min=98131, '
                                             'max=98526, per=100.00%, '
                                             'avg=98347.38, stdev=105.43, '
                                             'samples=13\n'
                                             '   iops        : min=196262, '
                                             'max=197052, avg=196694.85, '
                                             'stdev=210.89, samples=13\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=59.60%, 20=40.24%, '
                                             '50=0.12%\n'
                                             '  lat (usec)   : 100=0.03%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=18.67%, '
                                             'sys=52.14%, ctx=689064, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1376451,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=96.0MiB/s (101MB/s), '
                                             '96.0MiB/s-96.0MiB/s '
                                             '(101MB/s-101MB/s), io=672MiB '
                                             '(705MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=1979948/0, '
                                             'merge=0/0, ticks=10785/0, '
                                             'in_queue=10785, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.output'),
                                   'state': {'begin': 1666744478.2776206,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.796734094619751,
                                             'end': 1666744479.0743546,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_135.state')},
                       'cmd_136': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.output'),
                                   'state': {'begin': 1666744481.1065567,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7901074886322021,
                                             'end': 1666744481.8966641,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_136.state')},
                       'cmd_137': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.output'),
                                   'state': {'begin': 1666744481.898761,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.471136093139648,
                                             'end': 1666744493.3698971,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_137.state')},
                       'cmd_138': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751582,\n'
                                             '  "timestamp_ms" : '
                                             '1666751582489,\n'
                                             '  "time" : "Wed Oct 26 04:33:02 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 814672896,\n'
                                             '        "io_kbytes" : 795579,\n'
                                             '        "bw_bytes" : 116365218,\n'
                                             '        "bw" : 113637,\n'
                                             '        "iops" : 227275.674904,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1591157,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 17412,\n'
                                             '          "mean" : 151.050024,\n'
                                             '          "stddev" : 62.073951,\n'
                                             '          "N" : 1591157\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5881,\n'
                                             '          "max" : 120396,\n'
                                             '          "mean" : 8186.654414,\n'
                                             '          "stddev" : '
                                             '1593.597298,\n'
                                             '          "N" : 1591157,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7328,\n'
                                             '            "5.000000" : 7648,\n'
                                             '            "10.000000" : 7712,\n'
                                             '            "20.000000" : 7776,\n'
                                             '            "30.000000" : 7840,\n'
                                             '            "40.000000" : 7904,\n'
                                             '            "50.000000" : 7968,\n'
                                             '            "60.000000" : 8032,\n'
                                             '            "70.000000" : 8096,\n'
                                             '            "80.000000" : 8160,\n'
                                             '            "90.000000" : 8256,\n'
                                             '            "95.000000" : 8384,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '19840,\n'
                                             '            "99.900000" : '
                                             '20608,\n'
                                             '            "99.950000" : '
                                             '21120,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6071,\n'
                                             '          "max" : 120566,\n'
                                             '          "mean" : 8337.704438,\n'
                                             '          "stddev" : '
                                             '1596.070054,\n'
                                             '          "N" : 1591157\n'
                                             '        },\n'
                                             '        "bw_min" : 113483,\n'
                                             '        "bw_max" : 113870,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '113676.000000,\n'
                                             '        "bw_dev" : 107.724006,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 226966,\n'
                                             '        "iops_max" : 227740,\n'
                                             '        "iops_mean" : '
                                             '227352.071429,\n'
                                             '        "iops_stddev" : '
                                             '215.540274,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 33.528571,\n'
                                             '      "sys_cpu" : 66.442857,\n'
                                             '      "ctx" : 20,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.203818,\n'
                                             '        "2" : 50.796182,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 96.850279,\n'
                                             '        "20" : 2.729398,\n'
                                             '        "50" : 0.397949,\n'
                                             '        "100" : 0.022122,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=140637: Wed '
                                             'Oct 26 04:33:02 2022\n'
                                             '  read: IOPS=227k, BW=111MiB/s '
                                             '(116MB/s)(777MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=17412, avg=151.05, '
                                             'stdev=62.07\n'
                                             '    clat (usec): min=5, max=120, '
                                             'avg= 8.19, stdev= 1.59\n'
                                             '     lat (usec): min=6, max=120, '
                                             'avg= 8.34, stdev= 1.60\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7328],  '
                                             '5.00th=[ 7648], 10.00th=[ 7712], '
                                             '20.00th=[ 7776],\n'
                                             '     | 30.00th=[ 7840], '
                                             '40.00th=[ 7904], 50.00th=[ '
                                             '7968], 60.00th=[ 8032],\n'
                                             '     | 70.00th=[ 8096], '
                                             '80.00th=[ 8160], 90.00th=[ '
                                             '8256], 95.00th=[ 8384],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[19840], '
                                             '99.90th=[20608], '
                                             '99.95th=[21120],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=113483, '
                                             'max=113870, per=100.00%, '
                                             'avg=113676.00, stdev=107.72, '
                                             'samples=14\n'
                                             '   iops        : min=226966, '
                                             'max=227740, avg=227352.07, '
                                             'stdev=215.54, samples=14\n'
                                             '  lat (usec)   : 10=96.85%, '
                                             '20=2.73%, 50=0.40%, 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=33.53%, '
                                             'sys=66.44%, ctx=20, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=49.2%, '
                                             '2=50.8%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1591157,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=111MiB/s (116MB/s), '
                                             '111MiB/s-111MiB/s '
                                             '(116MB/s-116MB/s), io=777MiB '
                                             '(815MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.output'),
                                   'state': {'begin': 1666744493.3730302,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.787177324295044,
                                             'end': 1666744494.1602075,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_138.state')},
                       'cmd_139': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.output'),
                                   'state': {'begin': 1666744496.2266438,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7829077243804932,
                                             'end': 1666744497.0095515,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_139.state')},
                       'cmd_14': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.output'),
                                  'state': {'begin': 1666743894.412957,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring_cmd" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.06182074546814,
                                            'end': 1666743905.4747777,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_14.state')},
                       'cmd_140': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.output'),
                                   'state': {'begin': 1666744497.0118256,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.449482917785645,
                                             'end': 1666744508.4613085,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_140.state')},
                       'cmd_141': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751597,\n'
                                             '  "timestamp_ms" : '
                                             '1666751597582,\n'
                                             '  "time" : "Wed Oct 26 04:33:17 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 821842432,\n'
                                             '        "io_kbytes" : 802580,\n'
                                             '        "bw_bytes" : 117389291,\n'
                                             '        "bw" : 114637,\n'
                                             '        "iops" : 229275.817740,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1605160,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 17734,\n'
                                             '          "mean" : 149.779642,\n'
                                             '          "stddev" : 53.132110,\n'
                                             '          "N" : 1605160\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5751,\n'
                                             '          "max" : 97513,\n'
                                             '          "mean" : 8131.512809,\n'
                                             '          "stddev" : '
                                             '1581.103388,\n'
                                             '          "N" : 1605160,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7264,\n'
                                             '            "5.000000" : 7584,\n'
                                             '            "10.000000" : 7648,\n'
                                             '            "20.000000" : 7712,\n'
                                             '            "30.000000" : 7776,\n'
                                             '            "40.000000" : 7840,\n'
                                             '            "50.000000" : 7904,\n'
                                             '            "60.000000" : 7968,\n'
                                             '            "70.000000" : 8032,\n'
                                             '            "80.000000" : 8096,\n'
                                             '            "90.000000" : 8256,\n'
                                             '            "95.000000" : 8384,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '19840,\n'
                                             '            "99.900000" : '
                                             '20864,\n'
                                             '            "99.950000" : '
                                             '21120,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5952,\n'
                                             '          "max" : 97693,\n'
                                             '          "mean" : 8281.292451,\n'
                                             '          "stddev" : '
                                             '1583.644423,\n'
                                             '          "N" : 1605160\n'
                                             '        },\n'
                                             '        "bw_min" : 114397,\n'
                                             '        "bw_max" : 114917,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '114668.571429,\n'
                                             '        "bw_dev" : 150.823745,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 228794,\n'
                                             '        "iops_max" : 229834,\n'
                                             '        "iops_mean" : '
                                             '229337.214286,\n'
                                             '        "iops_stddev" : '
                                             '301.690666,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 35.814286,\n'
                                             '      "sys_cpu" : 64.157143,\n'
                                             '      "ctx" : 21,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.234718,\n'
                                             '        "2" : 50.765282,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 96.891587,\n'
                                             '        "20" : 2.622293,\n'
                                             '        "50" : 0.465873,\n'
                                             '        "100" : 0.020247,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=140686: Wed '
                                             'Oct 26 04:33:17 2022\n'
                                             '  read: IOPS=229k, BW=112MiB/s '
                                             '(117MB/s)(784MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=17734, avg=149.78, '
                                             'stdev=53.13\n'
                                             '    clat (nsec): min=5751, '
                                             'max=97513, avg=8131.51, '
                                             'stdev=1581.10\n'
                                             '     lat (nsec): min=5952, '
                                             'max=97693, avg=8281.29, '
                                             'stdev=1583.64\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7264],  '
                                             '5.00th=[ 7584], 10.00th=[ 7648], '
                                             '20.00th=[ 7712],\n'
                                             '     | 30.00th=[ 7776], '
                                             '40.00th=[ 7840], 50.00th=[ '
                                             '7904], 60.00th=[ 7968],\n'
                                             '     | 70.00th=[ 8032], '
                                             '80.00th=[ 8096], 90.00th=[ '
                                             '8256], 95.00th=[ 8384],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[19840], '
                                             '99.90th=[20864], '
                                             '99.95th=[21120],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=114397, '
                                             'max=114917, per=100.00%, '
                                             'avg=114668.57, stdev=150.82, '
                                             'samples=14\n'
                                             '   iops        : min=228794, '
                                             'max=229834, avg=229337.21, '
                                             'stdev=301.69, samples=14\n'
                                             '  lat (usec)   : 10=96.89%, '
                                             '20=2.62%, 50=0.47%, 100=0.02%\n'
                                             '  cpu          : usr=35.81%, '
                                             'sys=64.16%, ctx=21, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=49.2%, '
                                             '2=50.8%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1605160,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=112MiB/s (117MB/s), '
                                             '112MiB/s-112MiB/s '
                                             '(117MB/s-117MB/s), io=784MiB '
                                             '(822MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.output'),
                                   'state': {'begin': 1666744508.4644117,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7934420108795166,
                                             'end': 1666744509.2578537,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_141.state')},
                       'cmd_142': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.output'),
                                   'state': {'begin': 1666744511.3155067,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7826521396636963,
                                             'end': 1666744512.0981588,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_142.state')},
                       'cmd_143': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.output'),
                                   'state': {'begin': 1666744512.1005015,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.449578762054443,
                                             'end': 1666744523.5500803,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_143.state')},
                       'cmd_144': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751612,\n'
                                             '  "timestamp_ms" : '
                                             '1666751612671,\n'
                                             '  "time" : "Wed Oct 26 04:33:32 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 816369664,\n'
                                             '        "io_kbytes" : 797236,\n'
                                             '        "bw_bytes" : 116607579,\n'
                                             '        "bw" : 113874,\n'
                                             '        "iops" : 227749.035852,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1594471,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 23354,\n'
                                             '          "mean" : 150.289322,\n'
                                             '          "stddev" : 61.636434,\n'
                                             '          "N" : 1594471\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5841,\n'
                                             '          "max" : 119644,\n'
                                             '          "mean" : 8172.569133,\n'
                                             '          "stddev" : '
                                             '1601.463886,\n'
                                             '          "N" : 1594471,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7264,\n'
                                             '            "5.000000" : 7648,\n'
                                             '            "10.000000" : 7712,\n'
                                             '            "20.000000" : 7776,\n'
                                             '            "30.000000" : 7840,\n'
                                             '            "40.000000" : 7904,\n'
                                             '            "50.000000" : 7968,\n'
                                             '            "60.000000" : 8032,\n'
                                             '            "70.000000" : 8032,\n'
                                             '            "80.000000" : 8096,\n'
                                             '            "90.000000" : 8256,\n'
                                             '            "95.000000" : 8384,\n'
                                             '            "99.000000" : '
                                             '16512,\n'
                                             '            "99.500000" : '
                                             '19840,\n'
                                             '            "99.900000" : '
                                             '20864,\n'
                                             '            "99.950000" : '
                                             '21120,\n'
                                             '            "99.990000" : 61696\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6092,\n'
                                             '          "max" : 119814,\n'
                                             '          "mean" : 8322.858455,\n'
                                             '          "stddev" : '
                                             '1603.977324,\n'
                                             '          "N" : 1594471\n'
                                             '        },\n'
                                             '        "bw_min" : 113642,\n'
                                             '        "bw_max" : 114230,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '113906.642857,\n'
                                             '        "bw_dev" : 147.479650,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 227284,\n'
                                             '        "iops_max" : 228460,\n'
                                             '        "iops_mean" : '
                                             '227813.357143,\n'
                                             '        "iops_stddev" : '
                                             '295.000810,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.100000,\n'
                                             '      "sys_cpu" : 67.871429,\n'
                                             '      "ctx" : 27,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.172171,\n'
                                             '        "2" : 50.827829,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 96.864540,\n'
                                             '        "20" : 2.690359,\n'
                                             '        "50" : 0.421707,\n'
                                             '        "100" : 0.023142,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=140736: Wed '
                                             'Oct 26 04:33:32 2022\n'
                                             '  read: IOPS=228k, BW=111MiB/s '
                                             '(117MB/s)(779MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=23354, avg=150.29, '
                                             'stdev=61.64\n'
                                             '    clat (usec): min=5, max=119, '
                                             'avg= 8.17, stdev= 1.60\n'
                                             '     lat (usec): min=6, max=119, '
                                             'avg= 8.32, stdev= 1.60\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7264],  '
                                             '5.00th=[ 7648], 10.00th=[ 7712], '
                                             '20.00th=[ 7776],\n'
                                             '     | 30.00th=[ 7840], '
                                             '40.00th=[ 7904], 50.00th=[ '
                                             '7968], 60.00th=[ 8032],\n'
                                             '     | 70.00th=[ 8032], '
                                             '80.00th=[ 8096], 90.00th=[ '
                                             '8256], 95.00th=[ 8384],\n'
                                             '     | 99.00th=[16512], '
                                             '99.50th=[19840], '
                                             '99.90th=[20864], '
                                             '99.95th=[21120],\n'
                                             '     | 99.99th=[61696]\n'
                                             '   bw (  KiB/s): min=113642, '
                                             'max=114230, per=100.00%, '
                                             'avg=113906.64, stdev=147.48, '
                                             'samples=14\n'
                                             '   iops        : min=227284, '
                                             'max=228460, avg=227813.36, '
                                             'stdev=295.00, samples=14\n'
                                             '  lat (usec)   : 10=96.86%, '
                                             '20=2.69%, 50=0.42%, 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=32.10%, '
                                             'sys=67.87%, ctx=27, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=49.2%, '
                                             '2=50.8%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1594471,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=111MiB/s (117MB/s), '
                                             '111MiB/s-111MiB/s '
                                             '(117MB/s-117MB/s), io=779MiB '
                                             '(816MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.output'),
                                   'state': {'begin': 1666744523.5529957,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7888212203979492,
                                             'end': 1666744524.341817,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_144.state')},
                       'cmd_145': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.output'),
                                   'state': {'begin': 1666744526.4056482,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7824113368988037,
                                             'end': 1666744527.1880596,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_145.state')},
                       'cmd_146': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.output'),
                                   'state': {'begin': 1666744527.1904025,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.462955236434937,
                                             'end': 1666744538.6533577,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_146.state')},
                       'cmd_147': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751627,\n'
                                             '  "timestamp_ms" : '
                                             '1666751627772,\n'
                                             '  "time" : "Wed Oct 26 04:33:47 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 954444288,\n'
                                             '        "io_kbytes" : 932074,\n'
                                             '        "bw_bytes" : 136329708,\n'
                                             '        "bw" : 133134,\n'
                                             '        "iops" : 266268.675903,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1864147,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1552,\n'
                                             '          "max" : 23744,\n'
                                             '          "mean" : 2227.662047,\n'
                                             '          "stddev" : '
                                             '687.257728,\n'
                                             '          "N" : 1864148\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 241,\n'
                                             '          "max" : 115497,\n'
                                             '          "mean" : 5044.965217,\n'
                                             '          "stddev" : '
                                             '1900.368884,\n'
                                             '          "N" : 1864148,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4048,\n'
                                             '            "5.000000" : 4080,\n'
                                             '            "10.000000" : 4128,\n'
                                             '            "20.000000" : 4128,\n'
                                             '            "30.000000" : 4192,\n'
                                             '            "40.000000" : 4256,\n'
                                             '            "50.000000" : 5216,\n'
                                             '            "60.000000" : 5280,\n'
                                             '            "70.000000" : 5344,\n'
                                             '            "80.000000" : 5408,\n'
                                             '            "90.000000" : 5472,\n'
                                             '            "95.000000" : 5664,\n'
                                             '            "99.000000" : '
                                             '14272,\n'
                                             '            "99.500000" : '
                                             '18304,\n'
                                             '            "99.900000" : '
                                             '22400,\n'
                                             '            "99.950000" : '
                                             '22912,\n'
                                             '            "99.990000" : 60672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5591,\n'
                                             '          "max" : 117120,\n'
                                             '          "mean" : 7272.627264,\n'
                                             '          "stddev" : '
                                             '1826.947438,\n'
                                             '          "N" : 1864148\n'
                                             '        },\n'
                                             '        "bw_min" : 133014,\n'
                                             '        "bw_max" : 133288,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '133165.500000,\n'
                                             '        "bw_dev" : 95.984574,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 266028,\n'
                                             '        "iops_max" : 266576,\n'
                                             '        "iops_mean" : '
                                             '266331.000000,\n'
                                             '        "iops_stddev" : '
                                             '191.778318,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 33.700000,\n'
                                             '      "sys_cpu" : 66.271429,\n'
                                             '      "ctx" : 25,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.022799,\n'
                                             '        "10" : 97.619233,\n'
                                             '        "20" : 1.967710,\n'
                                             '        "50" : 0.368265,\n'
                                             '        "100" : 0.019741,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=140785: Wed '
                                             'Oct 26 04:33:47 2022\n'
                                             '  read: IOPS=266k, BW=130MiB/s '
                                             '(136MB/s)(910MiB/7001msec)\n'
                                             '    slat (nsec): min=1552, '
                                             'max=23744, avg=2227.66, '
                                             'stdev=687.26\n'
                                             '    clat (nsec): min=241, '
                                             'max=115497, avg=5044.97, '
                                             'stdev=1900.37\n'
                                             '     lat (usec): min=5, max=117, '
                                             'avg= 7.27, stdev= 1.83\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4048],  '
                                             '5.00th=[ 4080], 10.00th=[ 4128], '
                                             '20.00th=[ 4128],\n'
                                             '     | 30.00th=[ 4192], '
                                             '40.00th=[ 4256], 50.00th=[ '
                                             '5216], 60.00th=[ 5280],\n'
                                             '     | 70.00th=[ 5344], '
                                             '80.00th=[ 5408], 90.00th=[ '
                                             '5472], 95.00th=[ 5664],\n'
                                             '     | 99.00th=[14272], '
                                             '99.50th=[18304], '
                                             '99.90th=[22400], '
                                             '99.95th=[22912],\n'
                                             '     | 99.99th=[60672]\n'
                                             '   bw (  KiB/s): min=133014, '
                                             'max=133288, per=100.00%, '
                                             'avg=133165.50, stdev=95.98, '
                                             'samples=14\n'
                                             '   iops        : min=266028, '
                                             'max=266576, avg=266331.00, '
                                             'stdev=191.78, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.02%, '
                                             '10=97.62%, 20=1.97%, 50=0.37%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=33.70%, '
                                             'sys=66.27%, ctx=25, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1864147,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=130MiB/s (136MB/s), '
                                             '130MiB/s-130MiB/s '
                                             '(136MB/s-136MB/s), io=910MiB '
                                             '(954MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.output'),
                                   'state': {'begin': 1666744538.6566305,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7914872169494629,
                                             'end': 1666744539.4481177,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_147.state')},
                       'cmd_148': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.output'),
                                   'state': {'begin': 1666744541.479785,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7818942070007324,
                                             'end': 1666744542.2616792,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_148.state')},
                       'cmd_149': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.output'),
                                   'state': {'begin': 1666744542.2640674,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.468403339385986,
                                             'end': 1666744553.7324708,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_149.state')},
                       'cmd_15': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666750994,\n'
                                            '  "timestamp_ms" : '
                                            '1666750994677,\n'
                                            '  "time" : "Wed Oct 26 04:23:14 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring_cmd",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 285780992,\n'
                                            '        "io_kbytes" : 279083,\n'
                                            '        "bw_bytes" : 40820024,\n'
                                            '        "bw" : 39863,\n'
                                            '        "iops" : 79726.467647,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 558165,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2425,\n'
                                            '          "max" : 28102,\n'
                                            '          "mean" : 2654.688612,\n'
                                            '          "stddev" : 235.256413,\n'
                                            '          "N" : 558166\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 271,\n'
                                            '          "max" : 80301,\n'
                                            '          "mean" : 9230.039091,\n'
                                            '          "stddev" : '
                                            '1260.026978,\n'
                                            '          "N" : 558165,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8896,\n'
                                            '            "5.000000" : 8896,\n'
                                            '            "10.000000" : 8896,\n'
                                            '            "20.000000" : 9024,\n'
                                            '            "30.000000" : 9024,\n'
                                            '            "40.000000" : 9024,\n'
                                            '            "50.000000" : 9024,\n'
                                            '            "60.000000" : 9024,\n'
                                            '            "70.000000" : 9152,\n'
                                            '            "80.000000" : 9152,\n'
                                            '            "90.000000" : 9152,\n'
                                            '            "95.000000" : 9408,\n'
                                            '            "99.000000" : 15296,\n'
                                            '            "99.500000" : 16064,\n'
                                            '            "99.900000" : 19328,\n'
                                            '            "99.950000" : 27520,\n'
                                            '            "99.990000" : 55552\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9147,\n'
                                            '          "max" : 82936,\n'
                                            '          "mean" : 11884.727846,\n'
                                            '          "stddev" : '
                                            '1280.549295,\n'
                                            '          "N" : 558165\n'
                                            '        },\n'
                                            '        "bw_min" : 39774,\n'
                                            '        "bw_max" : 40040,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '39873.642857,\n'
                                            '        "bw_dev" : 66.964522,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 79548,\n'
                                            '        "iops_max" : 80080,\n'
                                            '        "iops_mean" : '
                                            '79747.285714,\n'
                                            '        "iops_stddev" : '
                                            '133.929045,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 13.842857,\n'
                                            '      "sys_cpu" : 45.942857,\n'
                                            '      "ctx" : 558046,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 97.336988,\n'
                                            '        "20" : 2.566087,\n'
                                            '        "50" : 0.071663,\n'
                                            '        "100" : 0.015587,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=138828: Wed '
                                            'Oct 26 04:23:14 2022\n'
                                            '  read: IOPS=79.7k, BW=38.9MiB/s '
                                            '(40.8MB/s)(273MiB/7001msec)\n'
                                            '    slat (nsec): min=2425, '
                                            'max=28102, avg=2654.69, '
                                            'stdev=235.26\n'
                                            '    clat (nsec): min=271, '
                                            'max=80301, avg=9230.04, '
                                            'stdev=1260.03\n'
                                            '     lat (nsec): min=9147, '
                                            'max=82936, avg=11884.73, '
                                            'stdev=1280.55\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8896],  5.00th=[ '
                                            '8896], 10.00th=[ 8896], 20.00th=[ '
                                            '9024],\n'
                                            '     | 30.00th=[ 9024], 40.00th=[ '
                                            '9024], 50.00th=[ 9024], 60.00th=[ '
                                            '9024],\n'
                                            '     | 70.00th=[ 9152], 80.00th=[ '
                                            '9152], 90.00th=[ 9152], 95.00th=[ '
                                            '9408],\n'
                                            '     | 99.00th=[15296], '
                                            '99.50th=[16064], 99.90th=[19328], '
                                            '99.95th=[27520],\n'
                                            '     | 99.99th=[55552]\n'
                                            '   bw (  KiB/s): min=39774, '
                                            'max=40040, per=100.00%, '
                                            'avg=39873.64, stdev=66.96, '
                                            'samples=14\n'
                                            '   iops        : min=79548, '
                                            'max=80080, avg=79747.29, '
                                            'stdev=133.93, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 10=97.34%, '
                                            '20=2.57%, 50=0.07%, 100=0.02%\n'
                                            '  cpu          : usr=13.84%, '
                                            'sys=45.94%, ctx=558046, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=558165,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=38.9MiB/s (40.8MB/s), '
                                            '38.9MiB/s-38.9MiB/s '
                                            '(40.8MB/s-40.8MB/s), io=273MiB '
                                            '(286MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.output'),
                                  'state': {'begin': 1666743905.4780238,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7868468761444092,
                                            'end': 1666743906.2648706,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_15.state')},
                       'cmd_150': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751642,\n'
                                             '  "timestamp_ms" : '
                                             '1666751642852,\n'
                                             '  "time" : "Wed Oct 26 04:34:02 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 955027456,\n'
                                             '        "io_kbytes" : 932644,\n'
                                             '        "bw_bytes" : 136413006,\n'
                                             '        "bw" : 133215,\n'
                                             '        "iops" : 266431.366948,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1865286,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1533,\n'
                                             '          "max" : 25778,\n'
                                             '          "mean" : 2232.692276,\n'
                                             '          "stddev" : '
                                             '689.704731,\n'
                                             '          "N" : 1865287\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 231,\n'
                                             '          "max" : 102753,\n'
                                             '          "mean" : 5046.237951,\n'
                                             '          "stddev" : '
                                             '1921.027764,\n'
                                             '          "N" : 1865287,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4048,\n'
                                             '            "5.000000" : 4080,\n'
                                             '            "10.000000" : 4128,\n'
                                             '            "20.000000" : 4128,\n'
                                             '            "30.000000" : 4192,\n'
                                             '            "40.000000" : 4256,\n'
                                             '            "50.000000" : 5216,\n'
                                             '            "60.000000" : 5280,\n'
                                             '            "70.000000" : 5344,\n'
                                             '            "80.000000" : 5408,\n'
                                             '            "90.000000" : 5472,\n'
                                             '            "95.000000" : 5664,\n'
                                             '            "99.000000" : '
                                             '14400,\n'
                                             '            "99.500000" : '
                                             '18304,\n'
                                             '            "99.900000" : '
                                             '22656,\n'
                                             '            "99.950000" : '
                                             '22912,\n'
                                             '            "99.990000" : 61696\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5651,\n'
                                             '          "max" : 104316,\n'
                                             '          "mean" : 7278.930227,\n'
                                             '          "stddev" : '
                                             '1850.805954,\n'
                                             '          "N" : 1865287\n'
                                             '        },\n'
                                             '        "bw_min" : 132892,\n'
                                             '        "bw_max" : 133392,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '133246.500000,\n'
                                             '        "bw_dev" : 157.042253,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 265784,\n'
                                             '        "iops_max" : 266785,\n'
                                             '        "iops_mean" : '
                                             '266493.214286,\n'
                                             '        "iops_stddev" : '
                                             '314.111734,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 35.300000,\n'
                                             '      "sys_cpu" : 64.671429,\n'
                                             '      "ctx" : 23,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.024286,\n'
                                             '        "10" : 97.589217,\n'
                                             '        "20" : 1.995404,\n'
                                             '        "50" : 0.367129,\n'
                                             '        "100" : 0.021820,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=140835: Wed '
                                             'Oct 26 04:34:02 2022\n'
                                             '  read: IOPS=266k, BW=130MiB/s '
                                             '(136MB/s)(911MiB/7001msec)\n'
                                             '    slat (nsec): min=1533, '
                                             'max=25778, avg=2232.69, '
                                             'stdev=689.70\n'
                                             '    clat (nsec): min=231, '
                                             'max=102753, avg=5046.24, '
                                             'stdev=1921.03\n'
                                             '     lat (usec): min=5, max=104, '
                                             'avg= 7.28, stdev= 1.85\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4048],  '
                                             '5.00th=[ 4080], 10.00th=[ 4128], '
                                             '20.00th=[ 4128],\n'
                                             '     | 30.00th=[ 4192], '
                                             '40.00th=[ 4256], 50.00th=[ '
                                             '5216], 60.00th=[ 5280],\n'
                                             '     | 70.00th=[ 5344], '
                                             '80.00th=[ 5408], 90.00th=[ '
                                             '5472], 95.00th=[ 5664],\n'
                                             '     | 99.00th=[14400], '
                                             '99.50th=[18304], '
                                             '99.90th=[22656], '
                                             '99.95th=[22912],\n'
                                             '     | 99.99th=[61696]\n'
                                             '   bw (  KiB/s): min=132892, '
                                             'max=133392, per=100.00%, '
                                             'avg=133246.50, stdev=157.04, '
                                             'samples=14\n'
                                             '   iops        : min=265784, '
                                             'max=266785, avg=266493.21, '
                                             'stdev=314.11, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.02%, '
                                             '10=97.59%, 20=2.00%, 50=0.37%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=35.30%, '
                                             'sys=64.67%, ctx=23, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1865286,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=130MiB/s (136MB/s), '
                                             '130MiB/s-130MiB/s '
                                             '(136MB/s-136MB/s), io=911MiB '
                                             '(955MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.output'),
                                   'state': {'begin': 1666744553.735333,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7919230461120605,
                                             'end': 1666744554.527256,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_150.state')},
                       'cmd_151': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.output'),
                                   'state': {'begin': 1666744556.5428421,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7913188934326172,
                                             'end': 1666744557.334161,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_151.state')},
                       'cmd_152': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.output'),
                                   'state': {'begin': 1666744557.3363864,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.450576782226562,
                                             'end': 1666744568.7869632,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_152.state')},
                       'cmd_153': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751657,\n'
                                             '  "timestamp_ms" : '
                                             '1666751657907,\n'
                                             '  "time" : "Wed Oct 26 04:34:17 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 949441536,\n'
                                             '        "io_kbytes" : 927189,\n'
                                             '        "bw_bytes" : 135615131,\n'
                                             '        "bw" : 132436,\n'
                                             '        "iops" : 264873.018140,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1854376,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1552,\n'
                                             '          "max" : 24926,\n'
                                             '          "mean" : 2245.204077,\n'
                                             '          "stddev" : '
                                             '687.051828,\n'
                                             '          "N" : 1854377\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 230,\n'
                                             '          "max" : 103073,\n'
                                             '          "mean" : 5066.601525,\n'
                                             '          "stddev" : '
                                             '1903.677678,\n'
                                             '          "N" : 1854377,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4080,\n'
                                             '            "5.000000" : 4128,\n'
                                             '            "10.000000" : 4128,\n'
                                             '            "20.000000" : 4192,\n'
                                             '            "30.000000" : 4192,\n'
                                             '            "40.000000" : 4256,\n'
                                             '            "50.000000" : 5216,\n'
                                             '            "60.000000" : 5280,\n'
                                             '            "70.000000" : 5408,\n'
                                             '            "80.000000" : 5472,\n'
                                             '            "90.000000" : 5536,\n'
                                             '            "95.000000" : 5664,\n'
                                             '            "99.000000" : '
                                             '14400,\n'
                                             '            "99.500000" : '
                                             '18304,\n'
                                             '            "99.900000" : '
                                             '22400,\n'
                                             '            "99.950000" : '
                                             '22912,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5641,\n'
                                             '          "max" : 104686,\n'
                                             '          "mean" : 7311.805603,\n'
                                             '          "stddev" : '
                                             '1830.883007,\n'
                                             '          "N" : 1854377\n'
                                             '        },\n'
                                             '        "bw_min" : 132278,\n'
                                             '        "bw_max" : 133014,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '132478.384615,\n'
                                             '        "bw_dev" : 208.302880,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 264556,\n'
                                             '        "iops_max" : 266028,\n'
                                             '        "iops_mean" : '
                                             '264957.000000,\n'
                                             '        "iops_stddev" : '
                                             '416.682533,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 30.785714,\n'
                                             '      "sys_cpu" : 69.185714,\n'
                                             '      "ctx" : 23,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.017310,\n'
                                             '        "10" : 97.592667,\n'
                                             '        "20" : 1.983039,\n'
                                             '        "50" : 0.384172,\n'
                                             '        "100" : 0.020600,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=140884: Wed '
                                             'Oct 26 04:34:17 2022\n'
                                             '  read: IOPS=265k, BW=129MiB/s '
                                             '(136MB/s)(905MiB/7001msec)\n'
                                             '    slat (nsec): min=1552, '
                                             'max=24926, avg=2245.20, '
                                             'stdev=687.05\n'
                                             '    clat (nsec): min=230, '
                                             'max=103073, avg=5066.60, '
                                             'stdev=1903.68\n'
                                             '     lat (usec): min=5, max=104, '
                                             'avg= 7.31, stdev= 1.83\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4080],  '
                                             '5.00th=[ 4128], 10.00th=[ 4128], '
                                             '20.00th=[ 4192],\n'
                                             '     | 30.00th=[ 4192], '
                                             '40.00th=[ 4256], 50.00th=[ '
                                             '5216], 60.00th=[ 5280],\n'
                                             '     | 70.00th=[ 5408], '
                                             '80.00th=[ 5472], 90.00th=[ '
                                             '5536], 95.00th=[ 5664],\n'
                                             '     | 99.00th=[14400], '
                                             '99.50th=[18304], '
                                             '99.90th=[22400], '
                                             '99.95th=[22912],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=132278, '
                                             'max=133014, per=100.00%, '
                                             'avg=132478.38, stdev=208.30, '
                                             'samples=13\n'
                                             '   iops        : min=264556, '
                                             'max=266028, avg=264957.00, '
                                             'stdev=416.68, samples=13\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.02%, '
                                             '10=97.59%, 20=1.98%, 50=0.38%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=30.79%, '
                                             'sys=69.19%, ctx=23, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1854376,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=129MiB/s (136MB/s), '
                                             '129MiB/s-129MiB/s '
                                             '(136MB/s-136MB/s), io=905MiB '
                                             '(949MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.output'),
                                   'state': {'begin': 1666744568.7901118,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8964223861694336,
                                             'end': 1666744569.6865342,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_153.state')},
                       'cmd_154': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.output'),
                                   'state': {'begin': 1666744570.6834624,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7792384624481201,
                                             'end': 1666744571.4627008,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_154.state')},
                       'cmd_155': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.output'),
                                   'state': {'begin': 1666744571.464903,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.05773639678955,
                                             'end': 1666744582.5226395,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_155.state')},
                       'cmd_156': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751671,\n'
                                             '  "timestamp_ms" : '
                                             '1666751671714,\n'
                                             '  "time" : "Wed Oct 26 04:34:31 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 764834304,\n'
                                             '        "io_kbytes" : 746908,\n'
                                             '        "bw_bytes" : 109246436,\n'
                                             '        "bw" : 106685,\n'
                                             '        "iops" : 213371.804028,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1493816,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1382,\n'
                                             '          "max" : 25729,\n'
                                             '          "mean" : 1532.744973,\n'
                                             '          "stddev" : '
                                             '144.972706,\n'
                                             '          "N" : 1493816\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 130,\n'
                                             '          "max" : 117480,\n'
                                             '          "mean" : 7572.991351,\n'
                                             '          "stddev" : '
                                             '2518.779534,\n'
                                             '          "N" : 1493817,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5152,\n'
                                             '            "5.000000" : 5152,\n'
                                             '            "10.000000" : 5216,\n'
                                             '            "20.000000" : 5280,\n'
                                             '            "30.000000" : 5344,\n'
                                             '            "40.000000" : 5472,\n'
                                             '            "50.000000" : 9024,\n'
                                             '            "60.000000" : 9024,\n'
                                             '            "70.000000" : 9152,\n'
                                             '            "80.000000" : 9280,\n'
                                             '            "90.000000" : 9408,\n'
                                             '            "95.000000" : '
                                             '10432,\n'
                                             '            "99.000000" : '
                                             '17024,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '19840,\n'
                                             '            "99.950000" : '
                                             '23168,\n'
                                             '            "99.990000" : 60160\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6092,\n'
                                             '          "max" : 119193,\n'
                                             '          "mean" : 9105.736391,\n'
                                             '          "stddev" : '
                                             '2595.087936,\n'
                                             '          "N" : 1493817\n'
                                             '        },\n'
                                             '        "bw_min" : 106453,\n'
                                             '        "bw_max" : 106968,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106711.785714,\n'
                                             '        "bw_dev" : 136.440000,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 212906,\n'
                                             '        "iops_max" : 213936,\n'
                                             '        "iops_mean" : '
                                             '213423.428571,\n'
                                             '        "iops_stddev" : '
                                             '272.960328,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.857143,\n'
                                             '      "sys_cpu" : 57.628571,\n'
                                             '      "ctx" : 747066,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 94.517866,\n'
                                             '        "20" : 5.375160,\n'
                                             '        "50" : 0.077787,\n'
                                             '        "100" : 0.020351,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=140924: Wed '
                                             'Oct 26 04:34:31 2022\n'
                                             '  read: IOPS=213k, BW=104MiB/s '
                                             '(109MB/s)(729MiB/7001msec)\n'
                                             '    slat (nsec): min=1382, '
                                             'max=25729, avg=1532.74, '
                                             'stdev=144.97\n'
                                             '    clat (nsec): min=130, '
                                             'max=117480, avg=7572.99, '
                                             'stdev=2518.78\n'
                                             '     lat (usec): min=6, max=119, '
                                             'avg= 9.11, stdev= 2.60\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5152],  '
                                             '5.00th=[ 5152], 10.00th=[ 5216], '
                                             '20.00th=[ 5280],\n'
                                             '     | 30.00th=[ 5344], '
                                             '40.00th=[ 5472], 50.00th=[ '
                                             '9024], 60.00th=[ 9024],\n'
                                             '     | 70.00th=[ 9152], '
                                             '80.00th=[ 9280], 90.00th=[ '
                                             '9408], 95.00th=[10432],\n'
                                             '     | 99.00th=[17024], '
                                             '99.50th=[18048], '
                                             '99.90th=[19840], '
                                             '99.95th=[23168],\n'
                                             '     | 99.99th=[60160]\n'
                                             '   bw (  KiB/s): min=106453, '
                                             'max=106968, per=100.00%, '
                                             'avg=106711.79, stdev=136.44, '
                                             'samples=14\n'
                                             '   iops        : min=212906, '
                                             'max=213936, avg=213423.43, '
                                             'stdev=272.96, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=94.52%, 20=5.38%, 50=0.08%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=19.86%, '
                                             'sys=57.63%, ctx=747066, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1493816,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=729MiB '
                                             '(765MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.output'),
                                   'state': {'begin': 1666744582.525979,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7881765365600586,
                                             'end': 1666744583.3141556,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_156.state')},
                       'cmd_157': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.output'),
                                   'state': {'begin': 1666744584.3125513,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7895863056182861,
                                             'end': 1666744585.1021376,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_157.state')},
                       'cmd_158': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.output'),
                                   'state': {'begin': 1666744585.1042175,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.063555002212524,
                                             'end': 1666744596.1677725,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_158.state')},
                       'cmd_159': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751685,\n'
                                             '  "timestamp_ms" : '
                                             '1666751685358,\n'
                                             '  "time" : "Wed Oct 26 04:34:45 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 765723648,\n'
                                             '        "io_kbytes" : 747777,\n'
                                             '        "bw_bytes" : 109373467,\n'
                                             '        "bw" : 106810,\n'
                                             '        "iops" : 213619.911441,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1495553,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1352,\n'
                                             '          "max" : 22452,\n'
                                             '          "mean" : 1493.274781,\n'
                                             '          "stddev" : '
                                             '148.492491,\n'
                                             '          "N" : 1495553\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 130,\n'
                                             '          "max" : 122009,\n'
                                             '          "mean" : 7620.069698,\n'
                                             '          "stddev" : '
                                             '2548.142265,\n'
                                             '          "N" : 1495554,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5152,\n'
                                             '            "5.000000" : 5216,\n'
                                             '            "10.000000" : 5216,\n'
                                             '            "20.000000" : 5344,\n'
                                             '            "30.000000" : 5408,\n'
                                             '            "40.000000" : 5536,\n'
                                             '            "50.000000" : 8896,\n'
                                             '            "60.000000" : 9024,\n'
                                             '            "70.000000" : 9024,\n'
                                             '            "80.000000" : 9152,\n'
                                             '            "90.000000" : 9408,\n'
                                             '            "95.000000" : '
                                             '10560,\n'
                                             '            "99.000000" : '
                                             '17280,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '20608,\n'
                                             '            "99.950000" : '
                                             '23168,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6161,\n'
                                             '          "max" : 123401,\n'
                                             '          "mean" : 9113.344539,\n'
                                             '          "stddev" : '
                                             '2616.635881,\n'
                                             '          "N" : 1495554\n'
                                             '        },\n'
                                             '        "bw_min" : 106727,\n'
                                             '        "bw_max" : 106938,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106835.214286,\n'
                                             '        "bw_dev" : 72.031167,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 213454,\n'
                                             '        "iops_max" : 213876,\n'
                                             '        "iops_mean" : '
                                             '213670.500000,\n'
                                             '        "iops_stddev" : '
                                             '144.097590,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 16.214286,\n'
                                             '      "sys_cpu" : 60.571429,\n'
                                             '      "ctx" : 748054,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 91.554763,\n'
                                             '        "20" : 8.329695,\n'
                                             '        "50" : 0.084517,\n'
                                             '        "100" : 0.016984,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=140964: Wed '
                                             'Oct 26 04:34:45 2022\n'
                                             '  read: IOPS=214k, BW=104MiB/s '
                                             '(109MB/s)(730MiB/7001msec)\n'
                                             '    slat (nsec): min=1352, '
                                             'max=22452, avg=1493.27, '
                                             'stdev=148.49\n'
                                             '    clat (nsec): min=130, '
                                             'max=122009, avg=7620.07, '
                                             'stdev=2548.14\n'
                                             '     lat (usec): min=6, max=123, '
                                             'avg= 9.11, stdev= 2.62\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5152],  '
                                             '5.00th=[ 5216], 10.00th=[ 5216], '
                                             '20.00th=[ 5344],\n'
                                             '     | 30.00th=[ 5408], '
                                             '40.00th=[ 5536], 50.00th=[ '
                                             '8896], 60.00th=[ 9024],\n'
                                             '     | 70.00th=[ 9024], '
                                             '80.00th=[ 9152], 90.00th=[ '
                                             '9408], 95.00th=[10560],\n'
                                             '     | 99.00th=[17280], '
                                             '99.50th=[18048], '
                                             '99.90th=[20608], '
                                             '99.95th=[23168],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=106727, '
                                             'max=106938, per=100.00%, '
                                             'avg=106835.21, stdev=72.03, '
                                             'samples=14\n'
                                             '   iops        : min=213454, '
                                             'max=213876, avg=213670.50, '
                                             'stdev=144.10, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=91.55%, 20=8.33%, 50=0.08%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=16.21%, '
                                             'sys=60.57%, ctx=748054, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1495553,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=730MiB '
                                             '(766MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.output'),
                                   'state': {'begin': 1666744596.170988,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7937331199645996,
                                             'end': 1666744596.9647212,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_159.state')},
                       'cmd_16': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.output'),
                                  'state': {'begin': 1666743907.2699497,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7967319488525391,
                                            'end': 1666743908.0666816,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_16.state')},
                       'cmd_160': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.output'),
                                   'state': {'begin': 1666744597.9525113,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7826642990112305,
                                             'end': 1666744598.7351756,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_160.state')},
                       'cmd_161': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.output'),
                                   'state': {'begin': 1666744598.7371712,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_io_uring-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.070218324661255,
                                             'end': 1666744609.8073895,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_161.state')},
                       'cmd_162': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751698,\n'
                                             '  "timestamp_ms" : '
                                             '1666751698998,\n'
                                             '  "time" : "Wed Oct 26 04:34:58 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 762477568,\n'
                                             '        "io_kbytes" : 744607,\n'
                                             '        "bw_bytes" : 108909808,\n'
                                             '        "bw" : 106357,\n'
                                             '        "iops" : 212714.326525,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1489213,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1352,\n'
                                             '          "max" : 26169,\n'
                                             '          "mean" : 1505.332639,\n'
                                             '          "stddev" : '
                                             '162.198529,\n'
                                             '          "N" : 1489213\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 140,\n'
                                             '          "max" : 121859,\n'
                                             '          "mean" : 7639.105499,\n'
                                             '          "stddev" : '
                                             '2552.195648,\n'
                                             '          "N" : 1489214,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 5088,\n'
                                             '            "5.000000" : 5152,\n'
                                             '            "10.000000" : 5216,\n'
                                             '            "20.000000" : 5280,\n'
                                             '            "30.000000" : 5344,\n'
                                             '            "40.000000" : 5600,\n'
                                             '            "50.000000" : 8896,\n'
                                             '            "60.000000" : 9024,\n'
                                             '            "70.000000" : 9024,\n'
                                             '            "80.000000" : 9152,\n'
                                             '            "90.000000" : 9664,\n'
                                             '            "95.000000" : '
                                             '10560,\n'
                                             '            "99.000000" : '
                                             '17280,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '20608,\n'
                                             '            "99.950000" : '
                                             '23168,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6061,\n'
                                             '          "max" : 123251,\n'
                                             '          "mean" : 9144.438204,\n'
                                             '          "stddev" : '
                                             '2629.067758,\n'
                                             '          "N" : 1489214\n'
                                             '        },\n'
                                             '        "bw_min" : 105852,\n'
                                             '        "bw_max" : 106636,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '106381.928571,\n'
                                             '        "bw_dev" : 196.151615,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 211705,\n'
                                             '        "iops_max" : 213272,\n'
                                             '        "iops_mean" : '
                                             '212763.928571,\n'
                                             '        "iops_stddev" : '
                                             '392.095448,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.128571,\n'
                                             '      "sys_cpu" : 59.557143,\n'
                                             '      "ctx" : 745014,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 90.268484,\n'
                                             '        "20" : 9.615884,\n'
                                             '        "50" : 0.085481,\n'
                                             '        "100" : 0.017727,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=141004: Wed '
                                             'Oct 26 04:34:58 2022\n'
                                             '  read: IOPS=213k, BW=104MiB/s '
                                             '(109MB/s)(727MiB/7001msec)\n'
                                             '    slat (nsec): min=1352, '
                                             'max=26169, avg=1505.33, '
                                             'stdev=162.20\n'
                                             '    clat (nsec): min=140, '
                                             'max=121859, avg=7639.11, '
                                             'stdev=2552.20\n'
                                             '     lat (usec): min=6, max=123, '
                                             'avg= 9.14, stdev= 2.63\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 5088],  '
                                             '5.00th=[ 5152], 10.00th=[ 5216], '
                                             '20.00th=[ 5280],\n'
                                             '     | 30.00th=[ 5344], '
                                             '40.00th=[ 5600], 50.00th=[ '
                                             '8896], 60.00th=[ 9024],\n'
                                             '     | 70.00th=[ 9024], '
                                             '80.00th=[ 9152], 90.00th=[ '
                                             '9664], 95.00th=[10560],\n'
                                             '     | 99.00th=[17280], '
                                             '99.50th=[18048], '
                                             '99.90th=[20608], '
                                             '99.95th=[23168],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=105852, '
                                             'max=106636, per=100.00%, '
                                             'avg=106381.93, stdev=196.15, '
                                             'samples=14\n'
                                             '   iops        : min=211705, '
                                             'max=213272, avg=212763.93, '
                                             'stdev=392.10, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=90.27%, 20=9.62%, 50=0.09%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=17.13%, '
                                             'sys=59.56%, ctx=745014, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1489213,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=104MiB/s (109MB/s), '
                                             '104MiB/s-104MiB/s '
                                             '(109MB/s-109MB/s), io=727MiB '
                                             '(762MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.output'),
                                   'state': {'begin': 1666744609.8105936,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7891902923583984,
                                             'end': 1666744610.599784,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_162.state')},
                       'cmd_163': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.output'),
                                   'state': {'begin': 1666744611.586182,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7797496318817139,
                                             'end': 1666744612.3659317,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_163.state')},
                       'cmd_164': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.output'),
                                   'state': {'begin': 1666744612.367877,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.070231676101685,
                                             'end': 1666744623.4381087,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_164.state')},
                       'cmd_165': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751712,\n'
                                             '  "timestamp_ms" : '
                                             '1666751712628,\n'
                                             '  "time" : "Wed Oct 26 04:35:12 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 779892224,\n'
                                             '        "io_kbytes" : 761613,\n'
                                             '        "bw_bytes" : 111397260,\n'
                                             '        "bw" : 108786,\n'
                                             '        "iops" : 217572.489644,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1523225,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1392,\n'
                                             '          "max" : 26299,\n'
                                             '          "mean" : 1533.941948,\n'
                                             '          "stddev" : '
                                             '145.787725,\n'
                                             '          "N" : 1523226\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 401,\n'
                                             '          "max" : 140834,\n'
                                             '          "mean" : 7415.475688,\n'
                                             '          "stddev" : '
                                             '1799.292095,\n'
                                             '          "N" : 1523226,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6880,\n'
                                             '            "5.000000" : 6944,\n'
                                             '            "10.000000" : 6944,\n'
                                             '            "20.000000" : 7008,\n'
                                             '            "30.000000" : 7072,\n'
                                             '            "40.000000" : 7072,\n'
                                             '            "50.000000" : 7136,\n'
                                             '            "60.000000" : 7200,\n'
                                             '            "70.000000" : 7264,\n'
                                             '            "80.000000" : 7328,\n'
                                             '            "90.000000" : 7392,\n'
                                             '            "95.000000" : 7648,\n'
                                             '            "99.000000" : '
                                             '15808,\n'
                                             '            "99.500000" : '
                                             '19328,\n'
                                             '            "99.900000" : '
                                             '21120,\n'
                                             '            "99.950000" : '
                                             '25472,\n'
                                             '            "99.990000" : 64768\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5911,\n'
                                             '          "max" : 142467,\n'
                                             '          "mean" : 8949.417635,\n'
                                             '          "stddev" : '
                                             '1803.451576,\n'
                                             '          "N" : 1523226\n'
                                             '        },\n'
                                             '        "bw_min" : 108527,\n'
                                             '        "bw_max" : 109183,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '108831.785714,\n'
                                             '        "bw_dev" : 172.703471,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 217054,\n'
                                             '        "iops_max" : 218366,\n'
                                             '        "iops_mean" : '
                                             '217663.714286,\n'
                                             '        "iops_stddev" : '
                                             '345.357881,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 23.371429,\n'
                                             '      "sys_cpu" : 54.428571,\n'
                                             '      "ctx" : 761236,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.033400,\n'
                                             '        "20" : 2.567119,\n'
                                             '        "50" : 0.367510,\n'
                                             '        "100" : 0.019761,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2179709,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 10715,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 10715,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=141045: Wed '
                                             'Oct 26 04:35:12 2022\n'
                                             '  read: IOPS=218k, BW=106MiB/s '
                                             '(111MB/s)(744MiB/7001msec)\n'
                                             '    slat (nsec): min=1392, '
                                             'max=26299, avg=1533.94, '
                                             'stdev=145.79\n'
                                             '    clat (nsec): min=401, '
                                             'max=140834, avg=7415.48, '
                                             'stdev=1799.29\n'
                                             '     lat (usec): min=5, max=142, '
                                             'avg= 8.95, stdev= 1.80\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6880],  '
                                             '5.00th=[ 6944], 10.00th=[ 6944], '
                                             '20.00th=[ 7008],\n'
                                             '     | 30.00th=[ 7072], '
                                             '40.00th=[ 7072], 50.00th=[ '
                                             '7136], 60.00th=[ 7200],\n'
                                             '     | 70.00th=[ 7264], '
                                             '80.00th=[ 7328], 90.00th=[ '
                                             '7392], 95.00th=[ 7648],\n'
                                             '     | 99.00th=[15808], '
                                             '99.50th=[19328], '
                                             '99.90th=[21120], '
                                             '99.95th=[25472],\n'
                                             '     | 99.99th=[64768]\n'
                                             '   bw (  KiB/s): min=108527, '
                                             'max=109183, per=100.00%, '
                                             'avg=108831.79, stdev=172.70, '
                                             'samples=14\n'
                                             '   iops        : min=217054, '
                                             'max=218366, avg=217663.71, '
                                             'stdev=345.36, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=97.03%, 20=2.57%, 50=0.37%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=23.37%, '
                                             'sys=54.43%, ctx=761236, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1523225,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=106MiB/s (111MB/s), '
                                             '106MiB/s-106MiB/s '
                                             '(111MB/s-111MB/s), io=744MiB '
                                             '(780MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2179709/0, '
                                             'merge=0/0, ticks=10715/0, '
                                             'in_queue=10715, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.output'),
                                   'state': {'begin': 1666744623.4409828,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.785879373550415,
                                             'end': 1666744624.2268622,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_165.state')},
                       'cmd_166': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.output'),
                                   'state': {'begin': 1666744625.21472,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7819104194641113,
                                             'end': 1666744625.9966304,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_166.state')},
                       'cmd_167': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.output'),
                                   'state': {'begin': 1666744625.9987113,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.078730583190918,
                                             'end': 1666744637.077442,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_167.state')},
                       'cmd_168': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751726,\n'
                                             '  "timestamp_ms" : '
                                             '1666751726267,\n'
                                             '  "time" : "Wed Oct 26 04:35:26 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 770420224,\n'
                                             '        "io_kbytes" : 752363,\n'
                                             '        "bw_bytes" : 110044311,\n'
                                             '        "bw" : 107465,\n'
                                             '        "iops" : 214930.009999,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1504725,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1412,\n'
                                             '          "max" : 15820,\n'
                                             '          "mean" : 1557.443289,\n'
                                             '          "stddev" : '
                                             '138.168896,\n'
                                             '          "N" : 1504726\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 400,\n'
                                             '          "max" : 123452,\n'
                                             '          "mean" : 7487.520184,\n'
                                             '          "stddev" : '
                                             '1736.521261,\n'
                                             '          "N" : 1504726,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6944,\n'
                                             '            "5.000000" : 7008,\n'
                                             '            "10.000000" : 7008,\n'
                                             '            "20.000000" : 7072,\n'
                                             '            "30.000000" : 7136,\n'
                                             '            "40.000000" : 7200,\n'
                                             '            "50.000000" : 7264,\n'
                                             '            "60.000000" : 7264,\n'
                                             '            "70.000000" : 7328,\n'
                                             '            "80.000000" : 7392,\n'
                                             '            "90.000000" : 7456,\n'
                                             '            "95.000000" : 7712,\n'
                                             '            "99.000000" : '
                                             '15680,\n'
                                             '            "99.500000" : '
                                             '19328,\n'
                                             '            "99.900000" : '
                                             '20608,\n'
                                             '            "99.950000" : '
                                             '24448,\n'
                                             '            "99.990000" : 64256\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5961,\n'
                                             '          "max" : 125115,\n'
                                             '          "mean" : 9044.963474,\n'
                                             '          "stddev" : '
                                             '1739.023859,\n'
                                             '          "N" : 1504726\n'
                                             '        },\n'
                                             '        "bw_min" : 107304,\n'
                                             '        "bw_max" : 107838,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '107519.384615,\n'
                                             '        "bw_dev" : 142.703970,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 214608,\n'
                                             '        "iops_max" : 215674,\n'
                                             '        "iops_mean" : '
                                             '215038.615385,\n'
                                             '        "iops_stddev" : '
                                             '285.036120,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 23.071429,\n'
                                             '      "sys_cpu" : 55.214286,\n'
                                             '      "ctx" : 751974,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.981841,\n'
                                             '        "20" : 2.622007,\n'
                                             '        "50" : 0.365117,\n'
                                             '        "100" : 0.019671,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2102056,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 10370,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 10370,\n'
                                             '      "util" : 99.085923\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=141084: Wed '
                                             'Oct 26 04:35:26 2022\n'
                                             '  read: IOPS=215k, BW=105MiB/s '
                                             '(110MB/s)(735MiB/7001msec)\n'
                                             '    slat (nsec): min=1412, '
                                             'max=15820, avg=1557.44, '
                                             'stdev=138.17\n'
                                             '    clat (nsec): min=400, '
                                             'max=123452, avg=7487.52, '
                                             'stdev=1736.52\n'
                                             '     lat (usec): min=5, max=125, '
                                             'avg= 9.04, stdev= 1.74\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6944],  '
                                             '5.00th=[ 7008], 10.00th=[ 7008], '
                                             '20.00th=[ 7072],\n'
                                             '     | 30.00th=[ 7136], '
                                             '40.00th=[ 7200], 50.00th=[ '
                                             '7264], 60.00th=[ 7264],\n'
                                             '     | 70.00th=[ 7328], '
                                             '80.00th=[ 7392], 90.00th=[ '
                                             '7456], 95.00th=[ 7712],\n'
                                             '     | 99.00th=[15680], '
                                             '99.50th=[19328], '
                                             '99.90th=[20608], '
                                             '99.95th=[24448],\n'
                                             '     | 99.99th=[64256]\n'
                                             '   bw (  KiB/s): min=107304, '
                                             'max=107838, per=100.00%, '
                                             'avg=107519.38, stdev=142.70, '
                                             'samples=13\n'
                                             '   iops        : min=214608, '
                                             'max=215674, avg=215038.62, '
                                             'stdev=285.04, samples=13\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.98%, 20=2.62%, 50=0.37%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=23.07%, '
                                             'sys=55.21%, ctx=751974, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1504725,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=105MiB/s (110MB/s), '
                                             '105MiB/s-105MiB/s '
                                             '(110MB/s-110MB/s), io=735MiB '
                                             '(770MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2102056/0, '
                                             'merge=0/0, ticks=10370/0, '
                                             'in_queue=10370, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.output'),
                                   'state': {'begin': 1666744637.0800567,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7916226387023926,
                                             'end': 1666744637.8716793,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_168.state')},
                       'cmd_169': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.output'),
                                   'state': {'begin': 1666744638.8652065,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7901461124420166,
                                             'end': 1666744639.6553526,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_169.state')},
                       'cmd_17': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.output'),
                                  'state': {'begin': 1666743908.0688016,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring_cmd" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.053016185760498,
                                            'end': 1666743919.1218178,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_17.state')},
                       'cmd_170': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.output'),
                                   'state': {'begin': 1666744639.65762,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.073652029037476,
                                             'end': 1666744650.731272,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_170.state')},
                       'cmd_171': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751739,\n'
                                             '  "timestamp_ms" : '
                                             '1666751739922,\n'
                                             '  "time" : "Wed Oct 26 04:35:39 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 771123712,\n'
                                             '        "io_kbytes" : 753050,\n'
                                             '        "bw_bytes" : 110144795,\n'
                                             '        "bw" : 107563,\n'
                                             '        "iops" : 215126.410513,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1506100,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1392,\n'
                                             '          "max" : 26369,\n'
                                             '          "mean" : 1556.186424,\n'
                                             '          "stddev" : '
                                             '150.779054,\n'
                                             '          "N" : 1506100\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 420,\n'
                                             '          "max" : 149550,\n'
                                             '          "mean" : 7484.908665,\n'
                                             '          "stddev" : '
                                             '1738.332903,\n'
                                             '          "N" : 1506101,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6944,\n'
                                             '            "5.000000" : 6944,\n'
                                             '            "10.000000" : 7008,\n'
                                             '            "20.000000" : 7072,\n'
                                             '            "30.000000" : 7136,\n'
                                             '            "40.000000" : 7200,\n'
                                             '            "50.000000" : 7264,\n'
                                             '            "60.000000" : 7264,\n'
                                             '            "70.000000" : 7328,\n'
                                             '            "80.000000" : 7392,\n'
                                             '            "90.000000" : 7520,\n'
                                             '            "95.000000" : 7776,\n'
                                             '            "99.000000" : '
                                             '15680,\n'
                                             '            "99.500000" : '
                                             '18816,\n'
                                             '            "99.900000" : '
                                             '20864,\n'
                                             '            "99.950000" : '
                                             '24960,\n'
                                             '            "99.990000" : 62208\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5911,\n'
                                             '          "max" : 151223,\n'
                                             '          "mean" : 9041.095007,\n'
                                             '          "stddev" : '
                                             '1740.860134,\n'
                                             '          "N" : 1506101\n'
                                             '        },\n'
                                             '        "bw_min" : 107410,\n'
                                             '        "bw_max" : 107798,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '107620.692308,\n'
                                             '        "bw_dev" : 114.054362,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 214820,\n'
                                             '        "iops_max" : 215596,\n'
                                             '        "iops_mean" : '
                                             '215241.230769,\n'
                                             '        "iops_stddev" : '
                                             '227.980319,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 25.185714,\n'
                                             '      "sys_cpu" : 52.814286,\n'
                                             '      "ctx" : 752769,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.987119,\n'
                                             '        "20" : 2.608924,\n'
                                             '        "50" : 0.373880,\n'
                                             '        "100" : 0.017197,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2105479,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 10382,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 10382,\n'
                                             '      "util" : 99.085923\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=141122: Wed '
                                             'Oct 26 04:35:39 2022\n'
                                             '  read: IOPS=215k, BW=105MiB/s '
                                             '(110MB/s)(735MiB/7001msec)\n'
                                             '    slat (nsec): min=1392, '
                                             'max=26369, avg=1556.19, '
                                             'stdev=150.78\n'
                                             '    clat (nsec): min=420, '
                                             'max=149550, avg=7484.91, '
                                             'stdev=1738.33\n'
                                             '     lat (usec): min=5, max=151, '
                                             'avg= 9.04, stdev= 1.74\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6944],  '
                                             '5.00th=[ 6944], 10.00th=[ 7008], '
                                             '20.00th=[ 7072],\n'
                                             '     | 30.00th=[ 7136], '
                                             '40.00th=[ 7200], 50.00th=[ '
                                             '7264], 60.00th=[ 7264],\n'
                                             '     | 70.00th=[ 7328], '
                                             '80.00th=[ 7392], 90.00th=[ '
                                             '7520], 95.00th=[ 7776],\n'
                                             '     | 99.00th=[15680], '
                                             '99.50th=[18816], '
                                             '99.90th=[20864], '
                                             '99.95th=[24960],\n'
                                             '     | 99.99th=[62208]\n'
                                             '   bw (  KiB/s): min=107410, '
                                             'max=107798, per=100.00%, '
                                             'avg=107620.69, stdev=114.05, '
                                             'samples=13\n'
                                             '   iops        : min=214820, '
                                             'max=215596, avg=215241.23, '
                                             'stdev=227.98, samples=13\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.99%, 20=2.61%, 50=0.37%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=25.19%, '
                                             'sys=52.81%, ctx=752769, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1506100,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=105MiB/s (110MB/s), '
                                             '105MiB/s-105MiB/s '
                                             '(110MB/s-110MB/s), io=735MiB '
                                             '(771MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2105479/0, '
                                             'merge=0/0, ticks=10382/0, '
                                             'in_queue=10382, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.output'),
                                   'state': {'begin': 1666744650.734316,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7895231246948242,
                                             'end': 1666744651.5238392,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_171.state')},
                       'cmd_172': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.output'),
                                   'state': {'begin': 1666744653.5630257,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.792830228805542,
                                             'end': 1666744654.355856,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_172.state')},
                       'cmd_173': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.output'),
                                   'state': {'begin': 1666744654.3583882,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_aio_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.524359941482544,
                                             'end': 1666744665.8827481,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_173.state')},
                       'cmd_174': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751754,\n'
                                             '  "timestamp_ms" : '
                                             '1666751754974,\n'
                                             '  "time" : "Wed Oct 26 04:35:54 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 925439488,\n'
                                             '        "io_kbytes" : 903749,\n'
                                             '        "bw_bytes" : 132186757,\n'
                                             '        "bw" : 129088,\n'
                                             '        "iops" : 258177.117555,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1807498,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1603,\n'
                                             '          "max" : 30868,\n'
                                             '          "mean" : 2315.003258,\n'
                                             '          "stddev" : '
                                             '734.458639,\n'
                                             '          "N" : 1807498\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 220,\n'
                                             '          "max" : 153418,\n'
                                             '          "mean" : 5163.300420,\n'
                                             '          "stddev" : '
                                             '1884.570066,\n'
                                             '          "N" : 1807499,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4128,\n'
                                             '            "5.000000" : 4128,\n'
                                             '            "10.000000" : 4192,\n'
                                             '            "20.000000" : 4192,\n'
                                             '            "30.000000" : 4256,\n'
                                             '            "40.000000" : 4384,\n'
                                             '            "50.000000" : 5344,\n'
                                             '            "60.000000" : 5408,\n'
                                             '            "70.000000" : 5536,\n'
                                             '            "80.000000" : 5600,\n'
                                             '            "90.000000" : 5792,\n'
                                             '            "95.000000" : 5920,\n'
                                             '            "99.000000" : '
                                             '13376,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '20608,\n'
                                             '            "99.950000" : '
                                             '20864,\n'
                                             '            "99.990000" : 63744\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5731,\n'
                                             '          "max" : 155091,\n'
                                             '          "mean" : 7478.303306,\n'
                                             '          "stddev" : '
                                             '1803.617973,\n'
                                             '          "N" : 1807499\n'
                                             '        },\n'
                                             '        "bw_min" : 128902,\n'
                                             '        "bw_max" : 129385,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '129123.500000,\n'
                                             '        "bw_dev" : 156.196006,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 257804,\n'
                                             '        "iops_max" : 258770,\n'
                                             '        "iops_mean" : '
                                             '258247.000000,\n'
                                             '        "iops_stddev" : '
                                             '312.392012,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 57.414286,\n'
                                             '      "sys_cpu" : 42.557143,\n'
                                             '      "ctx" : 23,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 1.656212,\n'
                                             '        "2" : 98.343788,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.012172,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.917453,\n'
                                             '        "20" : 1.799283,\n'
                                             '        "50" : 0.244924,\n'
                                             '        "100" : 0.017317,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=141171: Wed '
                                             'Oct 26 04:35:54 2022\n'
                                             '  read: IOPS=258k, BW=126MiB/s '
                                             '(132MB/s)(883MiB/7001msec)\n'
                                             '    slat (nsec): min=1603, '
                                             'max=30868, avg=2315.00, '
                                             'stdev=734.46\n'
                                             '    clat (nsec): min=220, '
                                             'max=153418, avg=5163.30, '
                                             'stdev=1884.57\n'
                                             '     lat (usec): min=5, max=155, '
                                             'avg= 7.48, stdev= 1.80\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4128],  '
                                             '5.00th=[ 4128], 10.00th=[ 4192], '
                                             '20.00th=[ 4192],\n'
                                             '     | 30.00th=[ 4256], '
                                             '40.00th=[ 4384], 50.00th=[ '
                                             '5344], 60.00th=[ 5408],\n'
                                             '     | 70.00th=[ 5536], '
                                             '80.00th=[ 5600], 90.00th=[ '
                                             '5792], 95.00th=[ 5920],\n'
                                             '     | 99.00th=[13376], '
                                             '99.50th=[18048], '
                                             '99.90th=[20608], '
                                             '99.95th=[20864],\n'
                                             '     | 99.99th=[63744]\n'
                                             '   bw (  KiB/s): min=128902, '
                                             'max=129385, per=100.00%, '
                                             'avg=129123.50, stdev=156.20, '
                                             'samples=14\n'
                                             '   iops        : min=257804, '
                                             'max=258770, avg=258247.00, '
                                             'stdev=312.39, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%, 750=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=97.92%, 20=1.80%, '
                                             '50=0.24%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=57.41%, '
                                             'sys=42.56%, ctx=23, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=1.7%, '
                                             '2=98.3%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1807498,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=126MiB/s (132MB/s), '
                                             '126MiB/s-126MiB/s '
                                             '(132MB/s-132MB/s), io=883MiB '
                                             '(925MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.output'),
                                   'state': {'begin': 1666744665.88565,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7863152027130127,
                                             'end': 1666744666.6719651,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_174.state')},
                       'cmd_175': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.output'),
                                   'state': {'begin': 1666744668.6855872,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7838649749755859,
                                             'end': 1666744669.4694521,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_175.state')},
                       'cmd_176': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.output'),
                                   'state': {'begin': 1666744669.4713986,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_aio_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.529305934906006,
                                             'end': 1666744681.0007045,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_176.state')},
                       'cmd_177': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751770,\n'
                                             '  "timestamp_ms" : '
                                             '1666751770080,\n'
                                             '  "time" : "Wed Oct 26 04:36:10 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 910187008,\n'
                                             '        "io_kbytes" : 888854,\n'
                                             '        "bw_bytes" : 130008142,\n'
                                             '        "bw" : 126961,\n'
                                             '        "iops" : 253921.868305,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1777707,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1593,\n'
                                             '          "max" : 30537,\n'
                                             '          "mean" : 2352.376895,\n'
                                             '          "stddev" : '
                                             '772.406286,\n'
                                             '          "N" : 1777708\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 241,\n'
                                             '          "max" : 135454,\n'
                                             '          "mean" : 5251.168977,\n'
                                             '          "stddev" : '
                                             '1853.294464,\n'
                                             '          "N" : 1777707,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4192,\n'
                                             '            "5.000000" : 4192,\n'
                                             '            "10.000000" : 4256,\n'
                                             '            "20.000000" : 4256,\n'
                                             '            "30.000000" : 4320,\n'
                                             '            "40.000000" : 4448,\n'
                                             '            "50.000000" : 5472,\n'
                                             '            "60.000000" : 5536,\n'
                                             '            "70.000000" : 5664,\n'
                                             '            "80.000000" : 5728,\n'
                                             '            "90.000000" : 5920,\n'
                                             '            "95.000000" : 5984,\n'
                                             '            "99.000000" : '
                                             '13376,\n'
                                             '            "99.500000" : '
                                             '17792,\n'
                                             '            "99.900000" : '
                                             '20608,\n'
                                             '            "99.950000" : '
                                             '20864,\n'
                                             '            "99.990000" : 60160\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5781,\n'
                                             '          "max" : 137148,\n'
                                             '          "mean" : 7603.546254,\n'
                                             '          "stddev" : '
                                             '1761.177099,\n'
                                             '          "N" : 1777707\n'
                                             '        },\n'
                                             '        "bw_min" : 126711,\n'
                                             '        "bw_max" : 127293,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '126994.142857,\n'
                                             '        "bw_dev" : 203.450942,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 253422,\n'
                                             '        "iops_max" : 254586,\n'
                                             '        "iops_mean" : '
                                             '253988.357143,\n'
                                             '        "iops_stddev" : '
                                             '407.005879,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 58.271429,\n'
                                             '      "sys_cpu" : 41.700000,\n'
                                             '      "ctx" : 20,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 1.653985,\n'
                                             '        "2" : 98.346015,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.014457,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.943981,\n'
                                             '        "20" : 1.808003,\n'
                                             '        "50" : 0.211565,\n'
                                             '        "100" : 0.018394,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=141222: Wed '
                                             'Oct 26 04:36:10 2022\n'
                                             '  read: IOPS=254k, BW=124MiB/s '
                                             '(130MB/s)(868MiB/7001msec)\n'
                                             '    slat (nsec): min=1593, '
                                             'max=30537, avg=2352.38, '
                                             'stdev=772.41\n'
                                             '    clat (nsec): min=241, '
                                             'max=135454, avg=5251.17, '
                                             'stdev=1853.29\n'
                                             '     lat (usec): min=5, max=137, '
                                             'avg= 7.60, stdev= 1.76\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4192],  '
                                             '5.00th=[ 4192], 10.00th=[ 4256], '
                                             '20.00th=[ 4256],\n'
                                             '     | 30.00th=[ 4320], '
                                             '40.00th=[ 4448], 50.00th=[ '
                                             '5472], 60.00th=[ 5536],\n'
                                             '     | 70.00th=[ 5664], '
                                             '80.00th=[ 5728], 90.00th=[ '
                                             '5920], 95.00th=[ 5984],\n'
                                             '     | 99.00th=[13376], '
                                             '99.50th=[17792], '
                                             '99.90th=[20608], '
                                             '99.95th=[20864],\n'
                                             '     | 99.99th=[60160]\n'
                                             '   bw (  KiB/s): min=126711, '
                                             'max=127293, per=100.00%, '
                                             'avg=126994.14, stdev=203.45, '
                                             'samples=14\n'
                                             '   iops        : min=253422, '
                                             'max=254586, avg=253988.36, '
                                             'stdev=407.01, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=97.94%, 20=1.81%, '
                                             '50=0.21%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=58.27%, '
                                             'sys=41.70%, ctx=20, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=1.7%, '
                                             '2=98.3%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1777707,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=124MiB/s (130MB/s), '
                                             '124MiB/s-124MiB/s '
                                             '(130MB/s-130MB/s), io=868MiB '
                                             '(910MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.output'),
                                   'state': {'begin': 1666744681.0033295,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7950851917266846,
                                             'end': 1666744681.7984147,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_177.state')},
                       'cmd_178': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.output'),
                                   'state': {'begin': 1666744683.82356,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7866497039794922,
                                             'end': 1666744684.6102097,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_178.state')},
                       'cmd_179': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.output'),
                                   'state': {'begin': 1666744684.6121955,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_aio_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.426987886428833,
                                             'end': 1666744696.0391834,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_179.state')},
                       'cmd_18': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751008,\n'
                                            '  "timestamp_ms" : '
                                            '1666751008324,\n'
                                            '  "time" : "Wed Oct 26 04:23:28 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring_cmd",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 291316224,\n'
                                            '        "io_kbytes" : 284488,\n'
                                            '        "bw_bytes" : 41610659,\n'
                                            '        "bw" : 40635,\n'
                                            '        "iops" : 81270.675618,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 568976,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2325,\n'
                                            '          "max" : 28083,\n'
                                            '          "mean" : 2529.571946,\n'
                                            '          "stddev" : 230.071012,\n'
                                            '          "N" : 568977\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 270,\n'
                                            '          "max" : 111159,\n'
                                            '          "mean" : 9106.443340,\n'
                                            '          "stddev" : '
                                            '1251.821492,\n'
                                            '          "N" : 568976,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8768,\n'
                                            '            "5.000000" : 8768,\n'
                                            '            "10.000000" : 8768,\n'
                                            '            "20.000000" : 8896,\n'
                                            '            "30.000000" : 8896,\n'
                                            '            "40.000000" : 8896,\n'
                                            '            "50.000000" : 8896,\n'
                                            '            "60.000000" : 8896,\n'
                                            '            "70.000000" : 9024,\n'
                                            '            "80.000000" : 9024,\n'
                                            '            "90.000000" : 9024,\n'
                                            '            "95.000000" : 9280,\n'
                                            '            "99.000000" : 15552,\n'
                                            '            "99.500000" : 16512,\n'
                                            '            "99.900000" : 19072,\n'
                                            '            "99.950000" : 21632,\n'
                                            '            "99.990000" : 55552\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 8857,\n'
                                            '          "max" : 113743,\n'
                                            '          "mean" : 11636.015347,\n'
                                            '          "stddev" : '
                                            '1270.086719,\n'
                                            '          "N" : 568976\n'
                                            '        },\n'
                                            '        "bw_min" : 40559,\n'
                                            '        "bw_max" : 40758,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '40646.000000,\n'
                                            '        "bw_dev" : 70.886692,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 81118,\n'
                                            '        "iops_max" : 81516,\n'
                                            '        "iops_mean" : '
                                            '81292.000000,\n'
                                            '        "iops_stddev" : '
                                            '141.773384,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 16.200000,\n'
                                            '      "sys_cpu" : 43.085714,\n'
                                            '      "ctx" : 568842,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.012479,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 97.371067,\n'
                                            '        "20" : 2.537365,\n'
                                            '        "50" : 0.064678,\n'
                                            '        "100" : 0.014236,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=138867: Wed '
                                            'Oct 26 04:23:28 2022\n'
                                            '  read: IOPS=81.3k, BW=39.7MiB/s '
                                            '(41.6MB/s)(278MiB/7001msec)\n'
                                            '    slat (nsec): min=2325, '
                                            'max=28083, avg=2529.57, '
                                            'stdev=230.07\n'
                                            '    clat (nsec): min=270, '
                                            'max=111159, avg=9106.44, '
                                            'stdev=1251.82\n'
                                            '     lat (usec): min=8, max=113, '
                                            'avg=11.64, stdev= 1.27\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8768],  5.00th=[ '
                                            '8768], 10.00th=[ 8768], 20.00th=[ '
                                            '8896],\n'
                                            '     | 30.00th=[ 8896], 40.00th=[ '
                                            '8896], 50.00th=[ 8896], 60.00th=[ '
                                            '8896],\n'
                                            '     | 70.00th=[ 9024], 80.00th=[ '
                                            '9024], 90.00th=[ 9024], 95.00th=[ '
                                            '9280],\n'
                                            '     | 99.00th=[15552], '
                                            '99.50th=[16512], 99.90th=[19072], '
                                            '99.95th=[21632],\n'
                                            '     | 99.99th=[55552]\n'
                                            '   bw (  KiB/s): min=40559, '
                                            'max=40758, per=100.00%, '
                                            'avg=40646.00, stdev=70.89, '
                                            'samples=14\n'
                                            '   iops        : min=81118, '
                                            'max=81516, avg=81292.00, '
                                            'stdev=141.77, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 10=97.37%, '
                                            '20=2.54%, 50=0.06%, 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=16.20%, '
                                            'sys=43.09%, ctx=568842, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=568976,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=39.7MiB/s (41.6MB/s), '
                                            '39.7MiB/s-39.7MiB/s '
                                            '(41.6MB/s-41.6MB/s), io=278MiB '
                                            '(291MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.output'),
                                  'state': {'begin': 1666743919.1248143,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8096821308135986,
                                            'end': 1666743919.9344964,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_18.state')},
                       'cmd_180': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751785,\n'
                                             '  "timestamp_ms" : '
                                             '1666751785122,\n'
                                             '  "time" : "Wed Oct 26 04:36:25 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 922618368,\n'
                                             '        "io_kbytes" : 900994,\n'
                                             '        "bw_bytes" : 131783797,\n'
                                             '        "bw" : 128695,\n'
                                             '        "iops" : 257389.944294,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1801987,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1583,\n'
                                             '          "max" : 29826,\n'
                                             '          "mean" : 2299.051626,\n'
                                             '          "stddev" : '
                                             '743.697493,\n'
                                             '          "N" : 1801988\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 230,\n'
                                             '          "max" : 138339,\n'
                                             '          "mean" : 5203.031010,\n'
                                             '          "stddev" : '
                                             '1872.544552,\n'
                                             '          "N" : 1801987,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4128,\n'
                                             '            "5.000000" : 4192,\n'
                                             '            "10.000000" : 4192,\n'
                                             '            "20.000000" : 4256,\n'
                                             '            "30.000000" : 4320,\n'
                                             '            "40.000000" : 4448,\n'
                                             '            "50.000000" : 5408,\n'
                                             '            "60.000000" : 5472,\n'
                                             '            "70.000000" : 5600,\n'
                                             '            "80.000000" : 5664,\n'
                                             '            "90.000000" : 5856,\n'
                                             '            "95.000000" : 5984,\n'
                                             '            "99.000000" : '
                                             '13376,\n'
                                             '            "99.500000" : '
                                             '18048,\n'
                                             '            "99.900000" : '
                                             '20864,\n'
                                             '            "99.950000" : '
                                             '21120,\n'
                                             '            "99.990000" : 61184\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5740,\n'
                                             '          "max" : 140022,\n'
                                             '          "mean" : 7502.082299,\n'
                                             '          "stddev" : '
                                             '1785.297905,\n'
                                             '          "N" : 1801987\n'
                                             '        },\n'
                                             '        "bw_min" : 128549,\n'
                                             '        "bw_max" : 129325,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '128728.785714,\n'
                                             '        "bw_dev" : 240.665161,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 257098,\n'
                                             '        "iops_max" : 258650,\n'
                                             '        "iops_mean" : '
                                             '257457.714286,\n'
                                             '        "iops_stddev" : '
                                             '481.220792,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 59.585714,\n'
                                             '      "sys_cpu" : 40.385714,\n'
                                             '      "ctx" : 23,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 1.640911,\n'
                                             '        "2" : 98.359089,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.011876,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.912249,\n'
                                             '        "20" : 1.800734,\n'
                                             '        "50" : 0.250335,\n'
                                             '        "100" : 0.017148,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=141271: Wed '
                                             'Oct 26 04:36:25 2022\n'
                                             '  read: IOPS=257k, BW=126MiB/s '
                                             '(132MB/s)(880MiB/7001msec)\n'
                                             '    slat (nsec): min=1583, '
                                             'max=29826, avg=2299.05, '
                                             'stdev=743.70\n'
                                             '    clat (nsec): min=230, '
                                             'max=138339, avg=5203.03, '
                                             'stdev=1872.54\n'
                                             '     lat (usec): min=5, max=140, '
                                             'avg= 7.50, stdev= 1.79\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4128],  '
                                             '5.00th=[ 4192], 10.00th=[ 4192], '
                                             '20.00th=[ 4256],\n'
                                             '     | 30.00th=[ 4320], '
                                             '40.00th=[ 4448], 50.00th=[ '
                                             '5408], 60.00th=[ 5472],\n'
                                             '     | 70.00th=[ 5600], '
                                             '80.00th=[ 5664], 90.00th=[ '
                                             '5856], 95.00th=[ 5984],\n'
                                             '     | 99.00th=[13376], '
                                             '99.50th=[18048], '
                                             '99.90th=[20864], '
                                             '99.95th=[21120],\n'
                                             '     | 99.99th=[61184]\n'
                                             '   bw (  KiB/s): min=128549, '
                                             'max=129325, per=100.00%, '
                                             'avg=128728.79, stdev=240.67, '
                                             'samples=14\n'
                                             '   iops        : min=257098, '
                                             'max=258650, avg=257457.71, '
                                             'stdev=481.22, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=97.91%, 20=1.80%, '
                                             '50=0.25%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=59.59%, '
                                             'sys=40.39%, ctx=23, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=1.6%, '
                                             '2=98.4%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1801987,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=126MiB/s (132MB/s), '
                                             '126MiB/s-126MiB/s '
                                             '(132MB/s-132MB/s), io=880MiB '
                                             '(923MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.output'),
                                   'state': {'begin': 1666744696.0420241,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7890415191650391,
                                             'end': 1666744696.8310657,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_180.state')},
                       'cmd_181': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.output'),
                                   'state': {'begin': 1666744698.8490248,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7922971248626709,
                                             'end': 1666744699.641322,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_181.state')},
                       'cmd_182': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.output'),
                                   'state': {'begin': 1666744699.6432898,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.539600133895874,
                                             'end': 1666744711.18289,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_182.state')},
                       'cmd_183': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751800,\n'
                                             '  "timestamp_ms" : '
                                             '1666751800261,\n'
                                             '  "time" : "Wed Oct 26 04:36:40 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 892137472,\n'
                                             '        "io_kbytes" : 871228,\n'
                                             '        "bw_bytes" : 127430005,\n'
                                             '        "bw" : 124443,\n'
                                             '        "iops" : 248886.444794,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1742454,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1593,\n'
                                             '          "max" : 19317,\n'
                                             '          "mean" : 2306.401335,\n'
                                             '          "stddev" : '
                                             '734.829566,\n'
                                             '          "N" : 1742455\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 591,\n'
                                             '          "max" : 121829,\n'
                                             '          "mean" : 5472.362839,\n'
                                             '          "stddev" : '
                                             '1813.157116,\n'
                                             '          "N" : 1742454,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4448,\n'
                                             '            "5.000000" : 4512,\n'
                                             '            "10.000000" : 4512,\n'
                                             '            "20.000000" : 4576,\n'
                                             '            "30.000000" : 4576,\n'
                                             '            "40.000000" : 4640,\n'
                                             '            "50.000000" : 5728,\n'
                                             '            "60.000000" : 5792,\n'
                                             '            "70.000000" : 5856,\n'
                                             '            "80.000000" : 5920,\n'
                                             '            "90.000000" : 5984,\n'
                                             '            "95.000000" : 6176,\n'
                                             '            "99.000000" : '
                                             '13248,\n'
                                             '            "99.500000" : '
                                             '17536,\n'
                                             '            "99.900000" : '
                                             '20352,\n'
                                             '            "99.950000" : '
                                             '20864,\n'
                                             '            "99.990000" : 62208\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5901,\n'
                                             '          "max" : 123492,\n'
                                             '          "mean" : 7778.763796,\n'
                                             '          "stddev" : '
                                             '1710.924115,\n'
                                             '          "N" : 1742454\n'
                                             '        },\n'
                                             '        "bw_min" : 124145,\n'
                                             '        "bw_max" : 124816,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '124484.384615,\n'
                                             '        "bw_dev" : 207.810626,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 248290,\n'
                                             '        "iops_max" : 249632,\n'
                                             '        "iops_mean" : '
                                             '248968.692308,\n'
                                             '        "iops_stddev" : '
                                             '415.787483,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.300000,\n'
                                             '      "sys_cpu" : 60.671429,\n'
                                             '      "ctx" : 18,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 1.483425,\n'
                                             '        "2" : 98.516575,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.013200,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 98.015098,\n'
                                             '        "20" : 1.779043,\n'
                                             '        "50" : 0.169072,\n'
                                             '        "100" : 0.019513,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=141320: Wed '
                                             'Oct 26 04:36:40 2022\n'
                                             '  read: IOPS=249k, BW=122MiB/s '
                                             '(127MB/s)(851MiB/7001msec)\n'
                                             '    slat (nsec): min=1593, '
                                             'max=19317, avg=2306.40, '
                                             'stdev=734.83\n'
                                             '    clat (nsec): min=591, '
                                             'max=121829, avg=5472.36, '
                                             'stdev=1813.16\n'
                                             '     lat (usec): min=5, max=123, '
                                             'avg= 7.78, stdev= 1.71\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4448],  '
                                             '5.00th=[ 4512], 10.00th=[ 4512], '
                                             '20.00th=[ 4576],\n'
                                             '     | 30.00th=[ 4576], '
                                             '40.00th=[ 4640], 50.00th=[ '
                                             '5728], 60.00th=[ 5792],\n'
                                             '     | 70.00th=[ 5856], '
                                             '80.00th=[ 5920], 90.00th=[ '
                                             '5984], 95.00th=[ 6176],\n'
                                             '     | 99.00th=[13248], '
                                             '99.50th=[17536], '
                                             '99.90th=[20352], '
                                             '99.95th=[20864],\n'
                                             '     | 99.99th=[62208]\n'
                                             '   bw (  KiB/s): min=124145, '
                                             'max=124816, per=100.00%, '
                                             'avg=124484.38, stdev=207.81, '
                                             'samples=13\n'
                                             '   iops        : min=248290, '
                                             'max=249632, avg=248968.69, '
                                             'stdev=415.79, samples=13\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=98.02%, 20=1.78%, '
                                             '50=0.17%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=39.30%, '
                                             'sys=60.67%, ctx=18, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=1.5%, '
                                             '2=98.5%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1742454,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=122MiB/s (127MB/s), '
                                             '122MiB/s-122MiB/s '
                                             '(127MB/s-127MB/s), io=851MiB '
                                             '(892MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.output'),
                                   'state': {'begin': 1666744711.1857076,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7863950729370117,
                                             'end': 1666744711.9721026,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_183.state')},
                       'cmd_184': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.output'),
                                   'state': {'begin': 1666744713.9869568,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7969717979431152,
                                             'end': 1666744714.7839286,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_184.state')},
                       'cmd_185': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.output'),
                                   'state': {'begin': 1666744714.7859266,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.573662757873535,
                                             'end': 1666744726.3595893,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_185.state')},
                       'cmd_186': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751815,\n'
                                             '  "timestamp_ms" : '
                                             '1666751815448,\n'
                                             '  "time" : "Wed Oct 26 04:36:55 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 888960000,\n'
                                             '        "io_kbytes" : 868125,\n'
                                             '        "bw_bytes" : 126976146,\n'
                                             '        "bw" : 124000,\n'
                                             '        "iops" : 248000.142837,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1736249,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1613,\n'
                                             '          "max" : 33963,\n'
                                             '          "mean" : 2325.591072,\n'
                                             '          "stddev" : '
                                             '759.135311,\n'
                                             '          "N" : 1736249\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 621,\n'
                                             '          "max" : 121107,\n'
                                             '          "mean" : 5472.580111,\n'
                                             '          "stddev" : '
                                             '1812.845915,\n'
                                             '          "N" : 1736250,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4256,\n'
                                             '            "5.000000" : 4320,\n'
                                             '            "10.000000" : 4512,\n'
                                             '            "20.000000" : 4576,\n'
                                             '            "30.000000" : 4640,\n'
                                             '            "40.000000" : 4704,\n'
                                             '            "50.000000" : 5472,\n'
                                             '            "60.000000" : 5856,\n'
                                             '            "70.000000" : 5920,\n'
                                             '            "80.000000" : 5984,\n'
                                             '            "90.000000" : 6048,\n'
                                             '            "95.000000" : 6240,\n'
                                             '            "99.000000" : '
                                             '13376,\n'
                                             '            "99.500000" : '
                                             '17280,\n'
                                             '            "99.900000" : '
                                             '20096,\n'
                                             '            "99.950000" : '
                                             '20608,\n'
                                             '            "99.990000" : 60672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5891,\n'
                                             '          "max" : 122811,\n'
                                             '          "mean" : 7798.170796,\n'
                                             '          "stddev" : '
                                             '1715.249598,\n'
                                             '          "N" : 1736250\n'
                                             '        },\n'
                                             '        "bw_min" : 123725,\n'
                                             '        "bw_max" : 124153,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '124029.571429,\n'
                                             '        "bw_dev" : 106.987213,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 247450,\n'
                                             '        "iops_max" : 248306,\n'
                                             '        "iops_mean" : '
                                             '248059.000000,\n'
                                             '        "iops_stddev" : '
                                             '214.096671,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 35.585714,\n'
                                             '      "sys_cpu" : 64.385714,\n'
                                             '      "ctx" : 24,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 1.473003,\n'
                                             '        "2" : 98.526997,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.012268,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 97.911532,\n'
                                             '        "20" : 1.955998,\n'
                                             '        "50" : 0.096991,\n'
                                             '        "100" : 0.019410,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=141369: Wed '
                                             'Oct 26 04:36:55 2022\n'
                                             '  read: IOPS=248k, BW=121MiB/s '
                                             '(127MB/s)(848MiB/7001msec)\n'
                                             '    slat (nsec): min=1613, '
                                             'max=33963, avg=2325.59, '
                                             'stdev=759.14\n'
                                             '    clat (nsec): min=621, '
                                             'max=121107, avg=5472.58, '
                                             'stdev=1812.85\n'
                                             '     lat (usec): min=5, max=122, '
                                             'avg= 7.80, stdev= 1.72\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4256],  '
                                             '5.00th=[ 4320], 10.00th=[ 4512], '
                                             '20.00th=[ 4576],\n'
                                             '     | 30.00th=[ 4640], '
                                             '40.00th=[ 4704], 50.00th=[ '
                                             '5472], 60.00th=[ 5856],\n'
                                             '     | 70.00th=[ 5920], '
                                             '80.00th=[ 5984], 90.00th=[ '
                                             '6048], 95.00th=[ 6240],\n'
                                             '     | 99.00th=[13376], '
                                             '99.50th=[17280], '
                                             '99.90th=[20096], '
                                             '99.95th=[20608],\n'
                                             '     | 99.99th=[60672]\n'
                                             '   bw (  KiB/s): min=123725, '
                                             'max=124153, per=100.00%, '
                                             'avg=124029.57, stdev=106.99, '
                                             'samples=14\n'
                                             '   iops        : min=247450, '
                                             'max=248306, avg=248059.00, '
                                             'stdev=214.10, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=97.91%, 20=1.96%, '
                                             '50=0.10%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=35.59%, '
                                             'sys=64.39%, ctx=24, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=1.5%, '
                                             '2=98.5%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1736249,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=121MiB/s (127MB/s), '
                                             '121MiB/s-121MiB/s '
                                             '(127MB/s-127MB/s), io=848MiB '
                                             '(889MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.output'),
                                   'state': {'begin': 1666744726.362394,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.793710470199585,
                                             'end': 1666744727.1561046,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_186.state')},
                       'cmd_187': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.output'),
                                   'state': {'begin': 1666744729.1706247,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7985272407531738,
                                             'end': 1666744729.969152,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_187.state')},
                       'cmd_188': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.output'),
                                   'state': {'begin': 1666744729.971215,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.529080152511597,
                                             'end': 1666744741.5002952,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_188.state')},
                       'cmd_189': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751830,\n'
                                             '  "timestamp_ms" : '
                                             '1666751830581,\n'
                                             '  "time" : "Wed Oct 26 04:37:10 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 886879232,\n'
                                             '        "io_kbytes" : 866093,\n'
                                             '        "bw_bytes" : 126678936,\n'
                                             '        "bw" : 123709,\n'
                                             '        "iops" : 247419.511498,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1732184,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1623,\n'
                                             '          "max" : 26189,\n'
                                             '          "mean" : 2344.885200,\n'
                                             '          "stddev" : '
                                             '746.431565,\n'
                                             '          "N" : 1732185\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 621,\n'
                                             '          "max" : 175389,\n'
                                             '          "mean" : 5470.346042,\n'
                                             '          "stddev" : '
                                             '1841.692980,\n'
                                             '          "N" : 1732184,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4256,\n'
                                             '            "5.000000" : 4320,\n'
                                             '            "10.000000" : 4512,\n'
                                             '            "20.000000" : 4576,\n'
                                             '            "30.000000" : 4640,\n'
                                             '            "40.000000" : 4704,\n'
                                             '            "50.000000" : 5536,\n'
                                             '            "60.000000" : 5856,\n'
                                             '            "70.000000" : 5920,\n'
                                             '            "80.000000" : 5984,\n'
                                             '            "90.000000" : 6048,\n'
                                             '            "95.000000" : 6240,\n'
                                             '            "99.000000" : '
                                             '13120,\n'
                                             '            "99.500000" : '
                                             '17024,\n'
                                             '            "99.900000" : '
                                             '20096,\n'
                                             '            "99.950000" : '
                                             '20352,\n'
                                             '            "99.990000" : 64768\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5921,\n'
                                             '          "max" : 177122,\n'
                                             '          "mean" : 7815.230866,\n'
                                             '          "stddev" : '
                                             '1740.065131,\n'
                                             '          "N" : 1732184\n'
                                             '        },\n'
                                             '        "bw_min" : 123454,\n'
                                             '        "bw_max" : 124192,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '123727.230769,\n'
                                             '        "bw_dev" : 248.715753,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 246908,\n'
                                             '        "iops_max" : 248384,\n'
                                             '        "iops_mean" : '
                                             '247454.461538,\n'
                                             '        "iops_stddev" : '
                                             '497.431505,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 37.471429,\n'
                                             '      "sys_cpu" : 62.500000,\n'
                                             '      "ctx" : 22,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 1.468666,\n'
                                             '        "2" : 98.531334,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010045,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 98.048995,\n'
                                             '        "20" : 1.830464,\n'
                                             '        "50" : 0.086019,\n'
                                             '        "100" : 0.019109,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=141419: Wed '
                                             'Oct 26 04:37:10 2022\n'
                                             '  read: IOPS=247k, BW=121MiB/s '
                                             '(127MB/s)(846MiB/7001msec)\n'
                                             '    slat (nsec): min=1623, '
                                             'max=26189, avg=2344.89, '
                                             'stdev=746.43\n'
                                             '    clat (nsec): min=621, '
                                             'max=175389, avg=5470.35, '
                                             'stdev=1841.69\n'
                                             '     lat (usec): min=5, max=177, '
                                             'avg= 7.82, stdev= 1.74\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4256],  '
                                             '5.00th=[ 4320], 10.00th=[ 4512], '
                                             '20.00th=[ 4576],\n'
                                             '     | 30.00th=[ 4640], '
                                             '40.00th=[ 4704], 50.00th=[ '
                                             '5536], 60.00th=[ 5856],\n'
                                             '     | 70.00th=[ 5920], '
                                             '80.00th=[ 5984], 90.00th=[ '
                                             '6048], 95.00th=[ 6240],\n'
                                             '     | 99.00th=[13120], '
                                             '99.50th=[17024], '
                                             '99.90th=[20096], '
                                             '99.95th=[20352],\n'
                                             '     | 99.99th=[64768]\n'
                                             '   bw (  KiB/s): min=123454, '
                                             'max=124192, per=100.00%, '
                                             'avg=123727.23, stdev=248.72, '
                                             'samples=13\n'
                                             '   iops        : min=246908, '
                                             'max=248384, avg=247454.46, '
                                             'stdev=497.43, samples=13\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=98.05%, 20=1.83%, '
                                             '50=0.09%\n'
                                             '  lat (usec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=37.47%, '
                                             'sys=62.50%, ctx=22, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=1.5%, '
                                             '2=98.5%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1732184,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=121MiB/s (127MB/s), '
                                             '121MiB/s-121MiB/s '
                                             '(127MB/s-127MB/s), io=846MiB '
                                             '(887MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.output'),
                                   'state': {'begin': 1666744741.5030072,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7861819267272949,
                                             'end': 1666744742.289189,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_189.state')},
                       'cmd_19': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.output'),
                                  'state': {'begin': 1666743921.9537325,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7813303470611572,
                                            'end': 1666743922.7350628,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_19.state')},
                       'cmd_190': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.output'),
                                   'state': {'begin': 1666744743.2813637,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7844269275665283,
                                             'end': 1666744744.0657907,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_190.state')},
                       'cmd_191': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.output'),
                                   'state': {'begin': 1666744744.0681014,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.10428261756897,
                                             'end': 1666744755.172384,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_191.state')},
                       'cmd_192': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751844,\n'
                                             '  "timestamp_ms" : '
                                             '1666751844361,\n'
                                             '  "time" : "Wed Oct 26 04:37:24 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 727684608,\n'
                                             '        "io_kbytes" : 710629,\n'
                                             '        "bw_bytes" : 103940095,\n'
                                             '        "bw" : 101503,\n'
                                             '        "iops" : 203007.713184,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1421257,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1472,\n'
                                             '          "max" : 38092,\n'
                                             '          "mean" : 1673.767642,\n'
                                             '          "stddev" : '
                                             '256.286057,\n'
                                             '          "N" : 1421258\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 471,\n'
                                             '          "max" : 160200,\n'
                                             '          "mean" : 7924.788124,\n'
                                             '          "stddev" : '
                                             '1919.052306,\n'
                                             '          "N" : 1421258,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4704,\n'
                                             '            "5.000000" : 7328,\n'
                                             '            "10.000000" : 7520,\n'
                                             '            "20.000000" : 7584,\n'
                                             '            "30.000000" : 7648,\n'
                                             '            "40.000000" : 7712,\n'
                                             '            "50.000000" : 7776,\n'
                                             '            "60.000000" : 7840,\n'
                                             '            "70.000000" : 7904,\n'
                                             '            "80.000000" : 7968,\n'
                                             '            "90.000000" : 8032,\n'
                                             '            "95.000000" : 8160,\n'
                                             '            "99.000000" : '
                                             '15552,\n'
                                             '            "99.500000" : '
                                             '19584,\n'
                                             '            "99.900000" : '
                                             '25216,\n'
                                             '            "99.950000" : '
                                             '25472,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5992,\n'
                                             '          "max" : 162054,\n'
                                             '          "mean" : 9598.555765,\n'
                                             '          "stddev" : '
                                             '1912.374899,\n'
                                             '          "N" : 1421258\n'
                                             '        },\n'
                                             '        "bw_min" : 101229,\n'
                                             '        "bw_max" : 101973,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '101532.571429,\n'
                                             '        "bw_dev" : 208.463136,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 202458,\n'
                                             '        "iops_max" : 203946,\n'
                                             '        "iops_mean" : '
                                             '203065.214286,\n'
                                             '        "iops_stddev" : '
                                             '416.847542,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 22.600000,\n'
                                             '      "sys_cpu" : 57.742857,\n'
                                             '      "ctx" : 681640,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.858907,\n'
                                             '        "20" : 2.760233,\n'
                                             '        "50" : 0.346454,\n'
                                             '        "100" : 0.023078,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=141458: Wed '
                                             'Oct 26 04:37:24 2022\n'
                                             '  read: IOPS=203k, BW=99.1MiB/s '
                                             '(104MB/s)(694MiB/7001msec)\n'
                                             '    slat (nsec): min=1472, '
                                             'max=38092, avg=1673.77, '
                                             'stdev=256.29\n'
                                             '    clat (nsec): min=471, '
                                             'max=160200, avg=7924.79, '
                                             'stdev=1919.05\n'
                                             '     lat (usec): min=5, max=162, '
                                             'avg= 9.60, stdev= 1.91\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4704],  '
                                             '5.00th=[ 7328], 10.00th=[ 7520], '
                                             '20.00th=[ 7584],\n'
                                             '     | 30.00th=[ 7648], '
                                             '40.00th=[ 7712], 50.00th=[ '
                                             '7776], 60.00th=[ 7840],\n'
                                             '     | 70.00th=[ 7904], '
                                             '80.00th=[ 7968], 90.00th=[ '
                                             '8032], 95.00th=[ 8160],\n'
                                             '     | 99.00th=[15552], '
                                             '99.50th=[19584], '
                                             '99.90th=[25216], '
                                             '99.95th=[25472],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=101229, '
                                             'max=101973, per=100.00%, '
                                             'avg=101532.57, stdev=208.46, '
                                             'samples=14\n'
                                             '   iops        : min=202458, '
                                             'max=203946, avg=203065.21, '
                                             'stdev=416.85, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%, '
                                             '750=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.86%, 20=2.76%, 50=0.35%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=22.60%, '
                                             'sys=57.74%, ctx=681640, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1421257,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=99.1MiB/s (104MB/s), '
                                             '99.1MiB/s-99.1MiB/s '
                                             '(104MB/s-104MB/s), io=694MiB '
                                             '(728MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.output'),
                                   'state': {'begin': 1666744755.1754644,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7867915630340576,
                                             'end': 1666744755.962256,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_192.state')},
                       'cmd_193': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.output'),
                                   'state': {'begin': 1666744756.9621143,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7869017124176025,
                                             'end': 1666744757.749016,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_193.state')},
                       'cmd_194': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.output'),
                                   'state': {'begin': 1666744757.7510684,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.106256008148193,
                                             'end': 1666744768.8573244,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_194.state')},
                       'cmd_195': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751858,\n'
                                             '  "timestamp_ms" : '
                                             '1666751858044,\n'
                                             '  "time" : "Wed Oct 26 04:37:38 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 724207104,\n'
                                             '        "io_kbytes" : 707233,\n'
                                             '        "bw_bytes" : 103443380,\n'
                                             '        "bw" : 101018,\n'
                                             '        "iops" : 202037.566062,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1414465,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1472,\n'
                                             '          "max" : 29145,\n'
                                             '          "mean" : 1659.259898,\n'
                                             '          "stddev" : '
                                             '223.201329,\n'
                                             '          "N" : 1414466\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 451,\n'
                                             '          "max" : 122610,\n'
                                             '          "mean" : 7996.446592,\n'
                                             '          "stddev" : '
                                             '1863.055498,\n'
                                             '          "N" : 1414466,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4896,\n'
                                             '            "5.000000" : 7520,\n'
                                             '            "10.000000" : 7584,\n'
                                             '            "20.000000" : 7584,\n'
                                             '            "30.000000" : 7648,\n'
                                             '            "40.000000" : 7712,\n'
                                             '            "50.000000" : 7840,\n'
                                             '            "60.000000" : 7840,\n'
                                             '            "70.000000" : 7904,\n'
                                             '            "80.000000" : 7968,\n'
                                             '            "90.000000" : 8032,\n'
                                             '            "95.000000" : 8256,\n'
                                             '            "99.000000" : '
                                             '15680,\n'
                                             '            "99.500000" : '
                                             '19584,\n'
                                             '            "99.900000" : '
                                             '25472,\n'
                                             '            "99.950000" : '
                                             '26240,\n'
                                             '            "99.990000" : 67072\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5992,\n'
                                             '          "max" : 124454,\n'
                                             '          "mean" : 9655.706490,\n'
                                             '          "stddev" : '
                                             '1864.008513,\n'
                                             '          "N" : 1414466\n'
                                             '        },\n'
                                             '        "bw_min" : 100860,\n'
                                             '        "bw_max" : 101305,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '101042.214286,\n'
                                             '        "bw_dev" : 134.849305,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 201720,\n'
                                             '        "iops_max" : 202610,\n'
                                             '        "iops_mean" : '
                                             '202084.285714,\n'
                                             '        "iops_stddev" : '
                                             '269.778094,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 22.514286,\n'
                                             '      "sys_cpu" : 57.471429,\n'
                                             '      "ctx" : 693903,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010888,\n'
                                             '        "10" : 96.823039,\n'
                                             '        "20" : 2.759206,\n'
                                             '        "50" : 0.380002,\n'
                                             '        "100" : 0.020432,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=141498: Wed '
                                             'Oct 26 04:37:38 2022\n'
                                             '  read: IOPS=202k, BW=98.7MiB/s '
                                             '(103MB/s)(691MiB/7001msec)\n'
                                             '    slat (nsec): min=1472, '
                                             'max=29145, avg=1659.26, '
                                             'stdev=223.20\n'
                                             '    clat (nsec): min=451, '
                                             'max=122610, avg=7996.45, '
                                             'stdev=1863.06\n'
                                             '     lat (usec): min=5, max=124, '
                                             'avg= 9.66, stdev= 1.86\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4896],  '
                                             '5.00th=[ 7520], 10.00th=[ 7584], '
                                             '20.00th=[ 7584],\n'
                                             '     | 30.00th=[ 7648], '
                                             '40.00th=[ 7712], 50.00th=[ '
                                             '7840], 60.00th=[ 7840],\n'
                                             '     | 70.00th=[ 7904], '
                                             '80.00th=[ 7968], 90.00th=[ '
                                             '8032], 95.00th=[ 8256],\n'
                                             '     | 99.00th=[15680], '
                                             '99.50th=[19584], '
                                             '99.90th=[25472], '
                                             '99.95th=[26240],\n'
                                             '     | 99.99th=[67072]\n'
                                             '   bw (  KiB/s): min=100860, '
                                             'max=101305, per=100.00%, '
                                             'avg=101042.21, stdev=134.85, '
                                             'samples=14\n'
                                             '   iops        : min=201720, '
                                             'max=202610, avg=202084.29, '
                                             'stdev=269.78, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%, '
                                             '750=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.82%, 20=2.76%, 50=0.38%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=22.51%, '
                                             'sys=57.47%, ctx=693903, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1414465,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=98.7MiB/s (103MB/s), '
                                             '98.7MiB/s-98.7MiB/s '
                                             '(103MB/s-103MB/s), io=691MiB '
                                             '(724MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.output'),
                                   'state': {'begin': 1666744768.8603215,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7948360443115234,
                                             'end': 1666744769.6551576,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_195.state')},
                       'cmd_196': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.output'),
                                   'state': {'begin': 1666744770.6479423,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7802488803863525,
                                             'end': 1666744771.4281912,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_196.state')},
                       'cmd_197': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.output'),
                                   'state': {'begin': 1666744771.4306922,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_2_libaio-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.090673923492432,
                                             'end': 1666744782.5213661,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_197.state')},
                       'cmd_198': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751871,\n'
                                             '  "timestamp_ms" : '
                                             '1666751871710,\n'
                                             '  "time" : "Wed Oct 26 04:37:51 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 724407808,\n'
                                             '        "io_kbytes" : 707429,\n'
                                             '        "bw_bytes" : 103472047,\n'
                                             '        "bw" : 101046,\n'
                                             '        "iops" : 202093.700900,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1414858,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1462,\n'
                                             '          "max" : 24455,\n'
                                             '          "mean" : 1634.391682,\n'
                                             '          "stddev" : '
                                             '230.781785,\n'
                                             '          "N" : 1414858\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 451,\n'
                                             '          "max" : 166613,\n'
                                             '          "mean" : 8011.332859,\n'
                                             '          "stddev" : '
                                             '1891.589306,\n'
                                             '          "N" : 1414859,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4960,\n'
                                             '            "5.000000" : 7520,\n'
                                             '            "10.000000" : 7584,\n'
                                             '            "20.000000" : 7648,\n'
                                             '            "30.000000" : 7712,\n'
                                             '            "40.000000" : 7776,\n'
                                             '            "50.000000" : 7840,\n'
                                             '            "60.000000" : 7904,\n'
                                             '            "70.000000" : 7904,\n'
                                             '            "80.000000" : 7968,\n'
                                             '            "90.000000" : 8032,\n'
                                             '            "95.000000" : 8256,\n'
                                             '            "99.000000" : '
                                             '15808,\n'
                                             '            "99.500000" : '
                                             '19584,\n'
                                             '            "99.900000" : '
                                             '25472,\n'
                                             '            "99.950000" : '
                                             '25984,\n'
                                             '            "99.990000" : 66048\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5981,\n'
                                             '          "max" : 168406,\n'
                                             '          "mean" : 9645.724596,\n'
                                             '          "stddev" : '
                                             '1891.318592,\n'
                                             '          "N" : 1414859\n'
                                             '        },\n'
                                             '        "bw_min" : 100815,\n'
                                             '        "bw_max" : 101425,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '101073.285714,\n'
                                             '        "bw_dev" : 170.786609,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 201630,\n'
                                             '        "iops_max" : 202851,\n'
                                             '        "iops_mean" : '
                                             '202146.642857,\n'
                                             '        "iops_stddev" : '
                                             '341.731700,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.114286,\n'
                                             '      "sys_cpu" : 59.771429,\n'
                                             '      "ctx" : 694144,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 96.841308,\n'
                                             '        "20" : 2.744304,\n'
                                             '        "50" : 0.377635,\n'
                                             '        "100" : 0.023041,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=141538: Wed '
                                             'Oct 26 04:37:51 2022\n'
                                             '  read: IOPS=202k, BW=98.7MiB/s '
                                             '(103MB/s)(691MiB/7001msec)\n'
                                             '    slat (nsec): min=1462, '
                                             'max=24455, avg=1634.39, '
                                             'stdev=230.78\n'
                                             '    clat (nsec): min=451, '
                                             'max=166613, avg=8011.33, '
                                             'stdev=1891.59\n'
                                             '     lat (usec): min=5, max=168, '
                                             'avg= 9.65, stdev= 1.89\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4960],  '
                                             '5.00th=[ 7520], 10.00th=[ 7584], '
                                             '20.00th=[ 7648],\n'
                                             '     | 30.00th=[ 7712], '
                                             '40.00th=[ 7776], 50.00th=[ '
                                             '7840], 60.00th=[ 7904],\n'
                                             '     | 70.00th=[ 7904], '
                                             '80.00th=[ 7968], 90.00th=[ '
                                             '8032], 95.00th=[ 8256],\n'
                                             '     | 99.00th=[15808], '
                                             '99.50th=[19584], '
                                             '99.90th=[25472], '
                                             '99.95th=[25984],\n'
                                             '     | 99.99th=[66048]\n'
                                             '   bw (  KiB/s): min=100815, '
                                             'max=101425, per=100.00%, '
                                             'avg=101073.29, stdev=170.79, '
                                             'samples=14\n'
                                             '   iops        : min=201630, '
                                             'max=202851, avg=202146.64, '
                                             'stdev=341.73, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%, '
                                             '750=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=96.84%, 20=2.74%, 50=0.38%, '
                                             '100=0.02%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=20.11%, '
                                             'sys=59.77%, ctx=694144, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1414858,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=98.7MiB/s (103MB/s), '
                                             '98.7MiB/s-98.7MiB/s '
                                             '(103MB/s-103MB/s), io=691MiB '
                                             '(724MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.output'),
                                   'state': {'begin': 1666744782.524606,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_2_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7909767627716064,
                                             'end': 1666744783.3155828,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_198.state')},
                       'cmd_199': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.output'),
                                   'state': {'begin': 1666744784.3118296,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.784106969833374,
                                             'end': 1666744785.0959365,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_199.state')},
                       'cmd_20': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.output'),
                                  'state': {'begin': 1666743922.7373698,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.47360110282898,
                                            'end': 1666743934.2109709,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_20.state')},
                       'cmd_200': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.output'),
                                   'state': {'begin': 1666744785.0982733,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.04575800895691,
                                             'end': 1666744796.1440313,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_200.state')},
                       'cmd_201': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751885,\n'
                                             '  "timestamp_ms" : '
                                             '1666751885332,\n'
                                             '  "time" : "Wed Oct 26 04:38:05 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1191164928,\n'
                                             '        "io_kbytes" : 1163247,\n'
                                             '        "bw_bytes" : 170142112,\n'
                                             '        "bw" : 166154,\n'
                                             '        "iops" : 332308.241680,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2326490,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1142,\n'
                                             '          "max" : 25858,\n'
                                             '          "mean" : 2197.265368,\n'
                                             '          "stddev" : '
                                             '1099.926498,\n'
                                             '          "N" : 2326491\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 80,\n'
                                             '          "max" : 165000,\n'
                                             '          "mean" : 9370.234935,\n'
                                             '          "stddev" : '
                                             '2562.392231,\n'
                                             '          "N" : 2326493,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6048,\n'
                                             '            "5.000000" : 7648,\n'
                                             '            "10.000000" : 8640,\n'
                                             '            "20.000000" : 8768,\n'
                                             '            "30.000000" : 8768,\n'
                                             '            "40.000000" : 8768,\n'
                                             '            "50.000000" : 8896,\n'
                                             '            "60.000000" : 8896,\n'
                                             '            "70.000000" : 9024,\n'
                                             '            "80.000000" : 9152,\n'
                                             '            "90.000000" : '
                                             '10304,\n'
                                             '            "95.000000" : '
                                             '11584,\n'
                                             '            "99.000000" : '
                                             '21888,\n'
                                             '            "99.500000" : '
                                             '24192,\n'
                                             '            "99.900000" : '
                                             '28800,\n'
                                             '            "99.950000" : '
                                             '34048,\n'
                                             '            "99.990000" : 70144\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5481,\n'
                                             '          "max" : 166182,\n'
                                             '          "mean" : '
                                             '11567.500667,\n'
                                             '          "stddev" : '
                                             '2627.540346,\n'
                                             '          "N" : 2326493\n'
                                             '        },\n'
                                             '        "bw_min" : 165820,\n'
                                             '        "bw_max" : 166583,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '166195.857143,\n'
                                             '        "bw_dev" : 234.095923,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 331640,\n'
                                             '        "iops_max" : 333166,\n'
                                             '        "iops_mean" : '
                                             '332391.785714,\n'
                                             '        "iops_stddev" : '
                                             '468.114085,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.371429,\n'
                                             '      "sys_cpu" : 82.142857,\n'
                                             '      "ctx" : 13656,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.302774,\n'
                                             '        "10" : 82.745681,\n'
                                             '        "20" : 15.383217,\n'
                                             '        "50" : 1.505831,\n'
                                             '        "100" : 0.039759,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=141577: Wed Oct 26 04:38:05 '
                                             '2022\n'
                                             '  read: IOPS=332k, BW=162MiB/s '
                                             '(170MB/s)(1136MiB/7001msec)\n'
                                             '    slat (nsec): min=1142, '
                                             'max=25858, avg=2197.27, '
                                             'stdev=1099.93\n'
                                             '    clat (nsec): min=80, '
                                             'max=165000, avg=9370.23, '
                                             'stdev=2562.39\n'
                                             '     lat (usec): min=5, max=166, '
                                             'avg=11.57, stdev= 2.63\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6048],  '
                                             '5.00th=[ 7648], 10.00th=[ 8640], '
                                             '20.00th=[ 8768],\n'
                                             '     | 30.00th=[ 8768], '
                                             '40.00th=[ 8768], 50.00th=[ '
                                             '8896], 60.00th=[ 8896],\n'
                                             '     | 70.00th=[ 9024], '
                                             '80.00th=[ 9152], '
                                             '90.00th=[10304], '
                                             '95.00th=[11584],\n'
                                             '     | 99.00th=[21888], '
                                             '99.50th=[24192], '
                                             '99.90th=[28800], '
                                             '99.95th=[34048],\n'
                                             '     | 99.99th=[70144]\n'
                                             '   bw (  KiB/s): min=165820, '
                                             'max=166583, per=100.00%, '
                                             'avg=166195.86, stdev=234.10, '
                                             'samples=14\n'
                                             '   iops        : min=331640, '
                                             'max=333166, avg=332391.79, '
                                             'stdev=468.11, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.30%, 10=82.75%, 20=15.38%, '
                                             '50=1.51%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=15.37%, '
                                             'sys=82.14%, ctx=13656, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2326490,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=162MiB/s (170MB/s), '
                                             '162MiB/s-162MiB/s '
                                             '(170MB/s-170MB/s), io=1136MiB '
                                             '(1191MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.output'),
                                   'state': {'begin': 1666744796.147,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7930381298065186,
                                             'end': 1666744796.9400382,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_201.state')},
                       'cmd_202': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.output'),
                                   'state': {'begin': 1666744797.928816,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8015997409820557,
                                             'end': 1666744798.7304158,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_202.state')},
                       'cmd_203': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.output'),
                                   'state': {'begin': 1666744798.7327921,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.038516759872437,
                                             'end': 1666744809.771309,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_203.state')},
                       'cmd_204': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751898,\n'
                                             '  "timestamp_ms" : '
                                             '1666751898959,\n'
                                             '  "time" : "Wed Oct 26 04:38:18 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1195248640,\n'
                                             '        "io_kbytes" : 1167235,\n'
                                             '        "bw_bytes" : 170725416,\n'
                                             '        "bw" : 166724,\n'
                                             '        "iops" : 333447.507499,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2334466,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1142,\n'
                                             '          "max" : 21070,\n'
                                             '          "mean" : 2160.286033,\n'
                                             '          "stddev" : '
                                             '1053.646322,\n'
                                             '          "N" : 2334467\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 70,\n'
                                             '          "max" : 172754,\n'
                                             '          "mean" : 9375.272461,\n'
                                             '          "stddev" : '
                                             '2567.897705,\n'
                                             '          "N" : 2334469,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6048,\n'
                                             '            "5.000000" : 7584,\n'
                                             '            "10.000000" : 8640,\n'
                                             '            "20.000000" : 8768,\n'
                                             '            "30.000000" : 8768,\n'
                                             '            "40.000000" : 8768,\n'
                                             '            "50.000000" : 8896,\n'
                                             '            "60.000000" : 8896,\n'
                                             '            "70.000000" : 8896,\n'
                                             '            "80.000000" : '
                                             '10048,\n'
                                             '            "90.000000" : '
                                             '10304,\n'
                                             '            "95.000000" : '
                                             '11584,\n'
                                             '            "99.000000" : '
                                             '21888,\n'
                                             '            "99.500000" : '
                                             '23936,\n'
                                             '            "99.900000" : '
                                             '28288,\n'
                                             '            "99.950000" : '
                                             '33536,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5440,\n'
                                             '          "max" : 173937,\n'
                                             '          "mean" : '
                                             '11535.557651,\n'
                                             '          "stddev" : '
                                             '2590.223019,\n'
                                             '          "N" : 2334469\n'
                                             '        },\n'
                                             '        "bw_min" : 166368,\n'
                                             '        "bw_max" : 167296,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '166781.142857,\n'
                                             '        "bw_dev" : 248.585493,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 332737,\n'
                                             '        "iops_max" : 334592,\n'
                                             '        "iops_mean" : '
                                             '333562.357143,\n'
                                             '        "iops_stddev" : '
                                             '497.043197,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.000000,\n'
                                             '      "sys_cpu" : 82.514286,\n'
                                             '      "ctx" : 13822,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.012894,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.289659,\n'
                                             '        "10" : 79.779187,\n'
                                             '        "20" : 18.447945,\n'
                                             '        "50" : 1.422681,\n'
                                             '        "100" : 0.036282,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=141615: Wed Oct 26 04:38:18 '
                                             '2022\n'
                                             '  read: IOPS=333k, BW=163MiB/s '
                                             '(171MB/s)(1140MiB/7001msec)\n'
                                             '    slat (nsec): min=1142, '
                                             'max=21070, avg=2160.29, '
                                             'stdev=1053.65\n'
                                             '    clat (nsec): min=70, '
                                             'max=172754, avg=9375.27, '
                                             'stdev=2567.90\n'
                                             '     lat (usec): min=5, max=173, '
                                             'avg=11.54, stdev= 2.59\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6048],  '
                                             '5.00th=[ 7584], 10.00th=[ 8640], '
                                             '20.00th=[ 8768],\n'
                                             '     | 30.00th=[ 8768], '
                                             '40.00th=[ 8768], 50.00th=[ '
                                             '8896], 60.00th=[ 8896],\n'
                                             '     | 70.00th=[ 8896], '
                                             '80.00th=[10048], '
                                             '90.00th=[10304], '
                                             '95.00th=[11584],\n'
                                             '     | 99.00th=[21888], '
                                             '99.50th=[23936], '
                                             '99.90th=[28288], '
                                             '99.95th=[33536],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=166368, '
                                             'max=167296, per=100.00%, '
                                             'avg=166781.14, stdev=248.59, '
                                             'samples=14\n'
                                             '   iops        : min=332737, '
                                             'max=334592, avg=333562.36, '
                                             'stdev=497.04, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.29%, 10=79.78%, 20=18.45%, '
                                             '50=1.42%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=15.00%, '
                                             'sys=82.51%, ctx=13822, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2334466,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=163MiB/s (171MB/s), '
                                             '163MiB/s-163MiB/s '
                                             '(171MB/s-171MB/s), io=1140MiB '
                                             '(1195MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.output'),
                                   'state': {'begin': 1666744809.7744377,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7913088798522949,
                                             'end': 1666744810.5657465,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_204.state')},
                       'cmd_205': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.output'),
                                   'state': {'begin': 1666744811.5587492,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7833278179168701,
                                             'end': 1666744812.342077,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_205.state')},
                       'cmd_206': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.output'),
                                   'state': {'begin': 1666744812.344384,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.040228605270386,
                                             'end': 1666744823.3846126,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_206.state')},
                       'cmd_207': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751912,\n'
                                             '  "timestamp_ms" : '
                                             '1666751912572,\n'
                                             '  "time" : "Wed Oct 26 04:38:32 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1193305600,\n'
                                             '        "io_kbytes" : 1165337,\n'
                                             '        "bw_bytes" : 170447878,\n'
                                             '        "bw" : 166453,\n'
                                             '        "iops" : 332905.442080,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2330671,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1142,\n'
                                             '          "max" : 25989,\n'
                                             '          "mean" : 2188.620048,\n'
                                             '          "stddev" : '
                                             '1059.038506,\n'
                                             '          "N" : 2330672\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 70,\n'
                                             '          "max" : 194865,\n'
                                             '          "mean" : 9353.569892,\n'
                                             '          "stddev" : '
                                             '2502.558563,\n'
                                             '          "N" : 2330674,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6048,\n'
                                             '            "5.000000" : 7584,\n'
                                             '            "10.000000" : 8640,\n'
                                             '            "20.000000" : 8768,\n'
                                             '            "30.000000" : 8768,\n'
                                             '            "40.000000" : 8768,\n'
                                             '            "50.000000" : 8896,\n'
                                             '            "60.000000" : 8896,\n'
                                             '            "70.000000" : 8896,\n'
                                             '            "80.000000" : 9280,\n'
                                             '            "90.000000" : '
                                             '10304,\n'
                                             '            "95.000000" : '
                                             '11584,\n'
                                             '            "99.000000" : '
                                             '21888,\n'
                                             '            "99.500000" : '
                                             '23680,\n'
                                             '            "99.900000" : '
                                             '27264,\n'
                                             '            "99.950000" : '
                                             '32640,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5451,\n'
                                             '          "max" : 196047,\n'
                                             '          "mean" : '
                                             '11542.189115,\n'
                                             '          "stddev" : '
                                             '2537.835066,\n'
                                             '          "N" : 2330674\n'
                                             '        },\n'
                                             '        "bw_min" : 166200,\n'
                                             '        "bw_max" : 167348,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '166499.928571,\n'
                                             '        "bw_dev" : 288.296713,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 332400,\n'
                                             '        "iops_max" : 334696,\n'
                                             '        "iops_mean" : '
                                             '332999.857143,\n'
                                             '        "iops_stddev" : '
                                             '576.593425,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 13.900000,\n'
                                             '      "sys_cpu" : 83.671429,\n'
                                             '      "ctx" : 13746,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.012014,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.293006,\n'
                                             '        "10" : 81.953008,\n'
                                             '        "20" : 16.269478,\n'
                                             '        "50" : 1.433021,\n'
                                             '        "100" : 0.029820,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=141653: Wed Oct 26 04:38:32 '
                                             '2022\n'
                                             '  read: IOPS=333k, BW=163MiB/s '
                                             '(170MB/s)(1138MiB/7001msec)\n'
                                             '    slat (nsec): min=1142, '
                                             'max=25989, avg=2188.62, '
                                             'stdev=1059.04\n'
                                             '    clat (nsec): min=70, '
                                             'max=194865, avg=9353.57, '
                                             'stdev=2502.56\n'
                                             '     lat (usec): min=5, max=196, '
                                             'avg=11.54, stdev= 2.54\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6048],  '
                                             '5.00th=[ 7584], 10.00th=[ 8640], '
                                             '20.00th=[ 8768],\n'
                                             '     | 30.00th=[ 8768], '
                                             '40.00th=[ 8768], 50.00th=[ '
                                             '8896], 60.00th=[ 8896],\n'
                                             '     | 70.00th=[ 8896], '
                                             '80.00th=[ 9280], '
                                             '90.00th=[10304], '
                                             '95.00th=[11584],\n'
                                             '     | 99.00th=[21888], '
                                             '99.50th=[23680], '
                                             '99.90th=[27264], '
                                             '99.95th=[32640],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=166200, '
                                             'max=167348, per=100.00%, '
                                             'avg=166499.93, stdev=288.30, '
                                             'samples=14\n'
                                             '   iops        : min=332400, '
                                             'max=334696, avg=332999.86, '
                                             'stdev=576.59, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.29%, 10=81.95%, 20=16.27%, '
                                             '50=1.43%\n'
                                             '  lat (usec)   : 100=0.03%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=13.90%, '
                                             'sys=83.67%, ctx=13746, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2330671,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=163MiB/s (170MB/s), '
                                             '163MiB/s-163MiB/s '
                                             '(170MB/s-170MB/s), io=1138MiB '
                                             '(1193MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.output'),
                                   'state': {'begin': 1666744823.3879497,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.786017894744873,
                                             'end': 1666744824.1739676,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_207.state')},
                       'cmd_208': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.output'),
                                   'state': {'begin': 1666744825.1965132,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7827303409576416,
                                             'end': 1666744825.9792435,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_208.state')},
                       'cmd_209': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.output'),
                                   'state': {'begin': 1666744825.9816153,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.062126398086548,
                                             'end': 1666744837.0437417,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_209.state')},
                       'cmd_21': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751023,\n'
                                            '  "timestamp_ms" : '
                                            '1666751023343,\n'
                                            '  "time" : "Wed Oct 26 04:23:43 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 477709312,\n'
                                            '        "io_kbytes" : 466513,\n'
                                            '        "bw_bytes" : 68234439,\n'
                                            '        "bw" : 66635,\n'
                                            '        "iops" : 133270.389944,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 933026,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 250,\n'
                                            '          "max" : 10320,\n'
                                            '          "mean" : 279.075861,\n'
                                            '          "stddev" : 23.844744,\n'
                                            '          "N" : 933026\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 250,\n'
                                            '          "max" : 119153,\n'
                                            '          "mean" : 7007.570928,\n'
                                            '          "stddev" : '
                                            '1331.921014,\n'
                                            '          "N" : 933026,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6624,\n'
                                            '            "5.000000" : 6688,\n'
                                            '            "10.000000" : 6752,\n'
                                            '            "20.000000" : 6752,\n'
                                            '            "30.000000" : 6752,\n'
                                            '            "40.000000" : 6816,\n'
                                            '            "50.000000" : 6816,\n'
                                            '            "60.000000" : 6880,\n'
                                            '            "70.000000" : 6880,\n'
                                            '            "80.000000" : 6944,\n'
                                            '            "90.000000" : 7008,\n'
                                            '            "95.000000" : 7136,\n'
                                            '            "99.000000" : 14400,\n'
                                            '            "99.500000" : 17024,\n'
                                            '            "99.900000" : 18048,\n'
                                            '            "99.950000" : 18560,\n'
                                            '            "99.990000" : 59136\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6392,\n'
                                            '          "max" : 119454,\n'
                                            '          "mean" : 7286.646789,\n'
                                            '          "stddev" : '
                                            '1332.043757,\n'
                                            '          "N" : 933026\n'
                                            '        },\n'
                                            '        "bw_min" : 66542,\n'
                                            '        "bw_max" : 66739,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '66649.214286,\n'
                                            '        "bw_dev" : 62.060986,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 133084,\n'
                                            '        "iops_max" : 133478,\n'
                                            '        "iops_mean" : '
                                            '133298.428571,\n'
                                            '        "iops_stddev" : '
                                            '124.121971,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 99.985714,\n'
                                            '      "sys_cpu" : 0.000000,\n'
                                            '      "ctx" : 27,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.287508,\n'
                                            '        "20" : 1.672622,\n'
                                            '        "50" : 0.025401,\n'
                                            '        "100" : 0.014148,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-bdev_xnvme: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=138918: Wed Oct 26 04:23:43 '
                                            '2022\n'
                                            '  read: IOPS=133k, BW=65.1MiB/s '
                                            '(68.2MB/s)(456MiB/7001msec)\n'
                                            '    slat (nsec): min=250, '
                                            'max=10320, avg=279.08, '
                                            'stdev=23.84\n'
                                            '    clat (nsec): min=250, '
                                            'max=119153, avg=7007.57, '
                                            'stdev=1331.92\n'
                                            '     lat (usec): min=6, max=119, '
                                            'avg= 7.29, stdev= 1.33\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6624],  5.00th=[ '
                                            '6688], 10.00th=[ 6752], 20.00th=[ '
                                            '6752],\n'
                                            '     | 30.00th=[ 6752], 40.00th=[ '
                                            '6816], 50.00th=[ 6816], 60.00th=[ '
                                            '6880],\n'
                                            '     | 70.00th=[ 6880], 80.00th=[ '
                                            '6944], 90.00th=[ 7008], 95.00th=[ '
                                            '7136],\n'
                                            '     | 99.00th=[14400], '
                                            '99.50th=[17024], 99.90th=[18048], '
                                            '99.95th=[18560],\n'
                                            '     | 99.99th=[59136]\n'
                                            '   bw (  KiB/s): min=66542, '
                                            'max=66739, per=100.00%, '
                                            'avg=66649.21, stdev=62.06, '
                                            'samples=14\n'
                                            '   iops        : min=133084, '
                                            'max=133478, avg=133298.43, '
                                            'stdev=124.12, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 10=98.29%, '
                                            '20=1.67%, 50=0.03%, 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=99.99%, '
                                            'sys=0.00%, ctx=27, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=933026,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=65.1MiB/s (68.2MB/s), '
                                            '65.1MiB/s-65.1MiB/s '
                                            '(68.2MB/s-68.2MB/s), io=456MiB '
                                            '(478MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.output'),
                                  'state': {'begin': 1666743934.213952,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7880222797393799,
                                            'end': 1666743935.0019743,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_21.state')},
                       'cmd_210': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751926,\n'
                                             '  "timestamp_ms" : '
                                             '1666751926230,\n'
                                             '  "time" : "Wed Oct 26 04:38:46 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1147879936,\n'
                                             '        "io_kbytes" : 1120976,\n'
                                             '        "bw_bytes" : 163959425,\n'
                                             '        "bw" : 160116,\n'
                                             '        "iops" : 320232.681046,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2241949,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1193,\n'
                                             '          "max" : 33182,\n'
                                             '          "mean" : 2542.839603,\n'
                                             '          "stddev" : '
                                             '751.011231,\n'
                                             '          "N" : 2241950\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 120,\n'
                                             '          "max" : 161323,\n'
                                             '          "mean" : 9665.963411,\n'
                                             '          "stddev" : '
                                             '2557.083231,\n'
                                             '          "N" : 2241952,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6368,\n'
                                             '            "5.000000" : 8032,\n'
                                             '            "10.000000" : 9024,\n'
                                             '            "20.000000" : 9024,\n'
                                             '            "30.000000" : 9152,\n'
                                             '            "40.000000" : 9152,\n'
                                             '            "50.000000" : 9280,\n'
                                             '            "60.000000" : 9280,\n'
                                             '            "70.000000" : 9280,\n'
                                             '            "80.000000" : 9408,\n'
                                             '            "90.000000" : '
                                             '10688,\n'
                                             '            "95.000000" : '
                                             '10944,\n'
                                             '            "99.000000" : '
                                             '22912,\n'
                                             '            "99.500000" : '
                                             '24704,\n'
                                             '            "99.900000" : '
                                             '28800,\n'
                                             '            "99.950000" : '
                                             '33536,\n'
                                             '            "99.990000" : 71168\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5630,\n'
                                             '          "max" : 162565,\n'
                                             '          "mean" : '
                                             '12208.801880,\n'
                                             '          "stddev" : '
                                             '2451.667220,\n'
                                             '          "N" : 2241952\n'
                                             '        },\n'
                                             '        "bw_min" : 159647,\n'
                                             '        "bw_max" : 160652,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '160159.214286,\n'
                                             '        "bw_dev" : 275.571065,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 319294,\n'
                                             '        "iops_max" : 321304,\n'
                                             '        "iops_mean" : '
                                             '320318.500000,\n'
                                             '        "iops_stddev" : '
                                             '551.411999,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 18.742857,\n'
                                             '      "sys_cpu" : 78.857143,\n'
                                             '      "ctx" : 12881,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.304556,\n'
                                             '        "10" : 85.096405,\n'
                                             '        "20" : 12.938698,\n'
                                             '        "50" : 1.610117,\n'
                                             '        "100" : 0.036888,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=141691: Wed '
                                             'Oct 26 04:38:46 2022\n'
                                             '  read: IOPS=320k, BW=156MiB/s '
                                             '(164MB/s)(1095MiB/7001msec)\n'
                                             '    slat (nsec): min=1193, '
                                             'max=33182, avg=2542.84, '
                                             'stdev=751.01\n'
                                             '    clat (nsec): min=120, '
                                             'max=161323, avg=9665.96, '
                                             'stdev=2557.08\n'
                                             '     lat (usec): min=5, max=162, '
                                             'avg=12.21, stdev= 2.45\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6368],  '
                                             '5.00th=[ 8032], 10.00th=[ 9024], '
                                             '20.00th=[ 9024],\n'
                                             '     | 30.00th=[ 9152], '
                                             '40.00th=[ 9152], 50.00th=[ '
                                             '9280], 60.00th=[ 9280],\n'
                                             '     | 70.00th=[ 9280], '
                                             '80.00th=[ 9408], '
                                             '90.00th=[10688], '
                                             '95.00th=[10944],\n'
                                             '     | 99.00th=[22912], '
                                             '99.50th=[24704], '
                                             '99.90th=[28800], '
                                             '99.95th=[33536],\n'
                                             '     | 99.99th=[71168]\n'
                                             '   bw (  KiB/s): min=159647, '
                                             'max=160652, per=100.00%, '
                                             'avg=160159.21, stdev=275.57, '
                                             'samples=14\n'
                                             '   iops        : min=319294, '
                                             'max=321304, avg=320318.50, '
                                             'stdev=551.41, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.30%, 10=85.10%, 20=12.94%, '
                                             '50=1.61%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=18.74%, '
                                             'sys=78.86%, ctx=12881, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2241949,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=156MiB/s (164MB/s), '
                                             '156MiB/s-156MiB/s '
                                             '(164MB/s-164MB/s), io=1095MiB '
                                             '(1148MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.output'),
                                   'state': {'begin': 1666744837.0469315,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.792938232421875,
                                             'end': 1666744837.8398697,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_210.state')},
                       'cmd_211': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.output'),
                                   'state': {'begin': 1666744838.8321114,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7809031009674072,
                                             'end': 1666744839.6130145,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_211.state')},
                       'cmd_212': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.output'),
                                   'state': {'begin': 1666744839.6151216,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.057931184768677,
                                             'end': 1666744850.6730528,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_212.state')},
                       'cmd_213': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751939,\n'
                                             '  "timestamp_ms" : '
                                             '1666751939860,\n'
                                             '  "time" : "Wed Oct 26 04:38:59 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1146219008,\n'
                                             '        "io_kbytes" : 1119354,\n'
                                             '        "bw_bytes" : 163722183,\n'
                                             '        "bw" : 159884,\n'
                                             '        "iops" : 319769.318669,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2238705,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1202,\n'
                                             '          "max" : 26811,\n'
                                             '          "mean" : 2540.398739,\n'
                                             '          "stddev" : '
                                             '740.364198,\n'
                                             '          "N" : 2238706\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 124784,\n'
                                             '          "mean" : 9685.766814,\n'
                                             '          "stddev" : '
                                             '2569.003445,\n'
                                             '          "N" : 2238708,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6432,\n'
                                             '            "5.000000" : 8096,\n'
                                             '            "10.000000" : 9024,\n'
                                             '            "20.000000" : 9024,\n'
                                             '            "30.000000" : 9152,\n'
                                             '            "40.000000" : 9152,\n'
                                             '            "50.000000" : 9280,\n'
                                             '            "60.000000" : 9280,\n'
                                             '            "70.000000" : 9280,\n'
                                             '            "80.000000" : 9408,\n'
                                             '            "90.000000" : '
                                             '10688,\n'
                                             '            "95.000000" : '
                                             '10944,\n'
                                             '            "99.000000" : '
                                             '23168,\n'
                                             '            "99.500000" : '
                                             '24704,\n'
                                             '            "99.900000" : '
                                             '29568,\n'
                                             '            "99.950000" : '
                                             '34048,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5580,\n'
                                             '          "max" : 126016,\n'
                                             '          "mean" : '
                                             '12226.165602,\n'
                                             '          "stddev" : '
                                             '2465.670806,\n'
                                             '          "N" : 2238708\n'
                                             '        },\n'
                                             '        "bw_min" : 159323,\n'
                                             '        "bw_max" : 160812,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '159930.214286,\n'
                                             '        "bw_dev" : 504.450833,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 318646,\n'
                                             '        "iops_max" : 321625,\n'
                                             '        "iops_mean" : '
                                             '319860.500000,\n'
                                             '        "iops_stddev" : '
                                             '1009.131595,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.500000,\n'
                                             '      "sys_cpu" : 82.100000,\n'
                                             '      "ctx" : 12883,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.293429,\n'
                                             '        "10" : 84.859327,\n'
                                             '        "20" : 13.183113,\n'
                                             '        "50" : 1.610619,\n'
                                             '        "100" : 0.040336,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=141729: Wed '
                                             'Oct 26 04:38:59 2022\n'
                                             '  read: IOPS=320k, BW=156MiB/s '
                                             '(164MB/s)(1093MiB/7001msec)\n'
                                             '    slat (nsec): min=1202, '
                                             'max=26811, avg=2540.40, '
                                             'stdev=740.36\n'
                                             '    clat (nsec): min=110, '
                                             'max=124784, avg=9685.77, '
                                             'stdev=2569.00\n'
                                             '     lat (usec): min=5, max=126, '
                                             'avg=12.23, stdev= 2.47\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6432],  '
                                             '5.00th=[ 8096], 10.00th=[ 9024], '
                                             '20.00th=[ 9024],\n'
                                             '     | 30.00th=[ 9152], '
                                             '40.00th=[ 9152], 50.00th=[ '
                                             '9280], 60.00th=[ 9280],\n'
                                             '     | 70.00th=[ 9280], '
                                             '80.00th=[ 9408], '
                                             '90.00th=[10688], '
                                             '95.00th=[10944],\n'
                                             '     | 99.00th=[23168], '
                                             '99.50th=[24704], '
                                             '99.90th=[29568], '
                                             '99.95th=[34048],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=159323, '
                                             'max=160812, per=100.00%, '
                                             'avg=159930.21, stdev=504.45, '
                                             'samples=14\n'
                                             '   iops        : min=318646, '
                                             'max=321625, avg=319860.50, '
                                             'stdev=1009.13, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.29%, 10=84.86%, 20=13.18%, '
                                             '50=1.61%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=15.50%, '
                                             'sys=82.10%, ctx=12883, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2238705,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=156MiB/s (164MB/s), '
                                             '156MiB/s-156MiB/s '
                                             '(164MB/s-164MB/s), io=1093MiB '
                                             '(1146MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.output'),
                                   'state': {'begin': 1666744850.6764238,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7867345809936523,
                                             'end': 1666744851.4631584,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_213.state')},
                       'cmd_214': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.output'),
                                   'state': {'begin': 1666744852.4581199,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7854704856872559,
                                             'end': 1666744853.2435904,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_214.state')},
                       'cmd_215': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.output'),
                                   'state': {'begin': 1666744853.245876,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.065338611602783,
                                             'end': 1666744864.3112147,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_215.state')},
                       'cmd_216': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751953,\n'
                                             '  "timestamp_ms" : '
                                             '1666751953498,\n'
                                             '  "time" : "Wed Oct 26 04:39:13 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1122532864,\n'
                                             '        "io_kbytes" : 1096223,\n'
                                             '        "bw_bytes" : 160338932,\n'
                                             '        "bw" : 156580,\n'
                                             '        "iops" : 313161.405513,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2192443,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1212,\n'
                                             '          "max" : 24466,\n'
                                             '          "mean" : 2454.161673,\n'
                                             '          "stddev" : '
                                             '961.853907,\n'
                                             '          "N" : 2192444\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 130,\n'
                                             '          "max" : 143108,\n'
                                             '          "mean" : 9983.846668,\n'
                                             '          "stddev" : '
                                             '2529.342162,\n'
                                             '          "N" : 2192446,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7648,\n'
                                             '            "5.000000" : 8160,\n'
                                             '            "10.000000" : 9280,\n'
                                             '            "20.000000" : 9280,\n'
                                             '            "30.000000" : 9408,\n'
                                             '            "40.000000" : 9408,\n'
                                             '            "50.000000" : 9536,\n'
                                             '            "60.000000" : 9536,\n'
                                             '            "70.000000" : 9536,\n'
                                             '            "80.000000" : '
                                             '10688,\n'
                                             '            "90.000000" : '
                                             '10944,\n'
                                             '            "95.000000" : '
                                             '11072,\n'
                                             '            "99.000000" : '
                                             '22912,\n'
                                             '            "99.500000" : '
                                             '24704,\n'
                                             '            "99.900000" : '
                                             '28544,\n'
                                             '            "99.950000" : '
                                             '33536,\n'
                                             '            "99.990000" : 71168\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5500,\n'
                                             '          "max" : 144361,\n'
                                             '          "mean" : '
                                             '12438.008588,\n'
                                             '          "stddev" : '
                                             '2576.508476,\n'
                                             '          "N" : 2192446\n'
                                             '        },\n'
                                             '        "bw_min" : 156329,\n'
                                             '        "bw_max" : 156855,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '156627.857143,\n'
                                             '        "bw_dev" : 180.425078,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 312658,\n'
                                             '        "iops_max" : 313710,\n'
                                             '        "iops_mean" : '
                                             '313255.714286,\n'
                                             '        "iops_stddev" : '
                                             '361.150177,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.500000,\n'
                                             '      "sys_cpu" : 77.171429,\n'
                                             '      "ctx" : 12638,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.095647,\n'
                                             '        "10" : 78.931676,\n'
                                             '        "20" : 19.299658,\n'
                                             '        "50" : 1.626907,\n'
                                             '        "100" : 0.036307,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=141767: Wed '
                                             'Oct 26 04:39:13 2022\n'
                                             '  read: IOPS=313k, BW=153MiB/s '
                                             '(160MB/s)(1071MiB/7001msec)\n'
                                             '    slat (nsec): min=1212, '
                                             'max=24466, avg=2454.16, '
                                             'stdev=961.85\n'
                                             '    clat (nsec): min=130, '
                                             'max=143108, avg=9983.85, '
                                             'stdev=2529.34\n'
                                             '     lat (usec): min=5, max=144, '
                                             'avg=12.44, stdev= 2.58\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7648],  '
                                             '5.00th=[ 8160], 10.00th=[ 9280], '
                                             '20.00th=[ 9280],\n'
                                             '     | 30.00th=[ 9408], '
                                             '40.00th=[ 9408], 50.00th=[ '
                                             '9536], 60.00th=[ 9536],\n'
                                             '     | 70.00th=[ 9536], '
                                             '80.00th=[10688], '
                                             '90.00th=[10944], '
                                             '95.00th=[11072],\n'
                                             '     | 99.00th=[22912], '
                                             '99.50th=[24704], '
                                             '99.90th=[28544], '
                                             '99.95th=[33536],\n'
                                             '     | 99.99th=[71168]\n'
                                             '   bw (  KiB/s): min=156329, '
                                             'max=156855, per=100.00%, '
                                             'avg=156627.86, stdev=180.43, '
                                             'samples=14\n'
                                             '   iops        : min=312658, '
                                             'max=313710, avg=313255.71, '
                                             'stdev=361.15, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.10%, 10=78.93%, 20=19.30%, '
                                             '50=1.63%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=20.50%, '
                                             'sys=77.17%, ctx=12638, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2192443,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=153MiB/s (160MB/s), '
                                             '153MiB/s-153MiB/s '
                                             '(160MB/s-160MB/s), io=1071MiB '
                                             '(1123MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.output'),
                                   'state': {'begin': 1666744864.3145394,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7896699905395508,
                                             'end': 1666744865.1042094,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_216.state')},
                       'cmd_217': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.output'),
                                   'state': {'begin': 1666744867.1236086,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7923183441162109,
                                             'end': 1666744867.915927,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_217.state')},
                       'cmd_218': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.output'),
                                   'state': {'begin': 1666744867.9183056,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.49340534210205,
                                             'end': 1666744879.411711,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_218.state')},
                       'cmd_219': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751968,\n'
                                             '  "timestamp_ms" : '
                                             '1666751968527,\n'
                                             '  "time" : "Wed Oct 26 04:39:28 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1505933312,\n'
                                             '        "io_kbytes" : 1470638,\n'
                                             '        "bw_bytes" : 215102601,\n'
                                             '        "bw" : 210061,\n'
                                             '        "iops" : 420121.839737,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2941273,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 210,\n'
                                             '          "max" : 28404,\n'
                                             '          "mean" : 272.227385,\n'
                                             '          "stddev" : 44.424831,\n'
                                             '          "N" : 2941273\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 360,\n'
                                             '          "max" : 125015,\n'
                                             '          "mean" : 9017.813720,\n'
                                             '          "stddev" : '
                                             '2595.881437,\n'
                                             '          "N" : 2941276,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6752,\n'
                                             '            "5.000000" : 7456,\n'
                                             '            "10.000000" : 7840,\n'
                                             '            "20.000000" : 7968,\n'
                                             '            "30.000000" : 8032,\n'
                                             '            "40.000000" : 8160,\n'
                                             '            "50.000000" : 8256,\n'
                                             '            "60.000000" : 9152,\n'
                                             '            "70.000000" : 9280,\n'
                                             '            "80.000000" : 9408,\n'
                                             '            "90.000000" : 9664,\n'
                                             '            "95.000000" : '
                                             '10560,\n'
                                             '            "99.000000" : '
                                             '20352,\n'
                                             '            "99.500000" : '
                                             '22144,\n'
                                             '            "99.900000" : '
                                             '29568,\n'
                                             '            "99.950000" : '
                                             '36096,\n'
                                             '            "99.990000" : 69120\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5751,\n'
                                             '          "max" : 125285,\n'
                                             '          "mean" : 9290.041100,\n'
                                             '          "stddev" : '
                                             '2596.346238,\n'
                                             '          "N" : 2941276\n'
                                             '        },\n'
                                             '        "bw_min" : 209329,\n'
                                             '        "bw_max" : 211028,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '210143.428571,\n'
                                             '        "bw_dev" : 467.370338,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 418658,\n'
                                             '        "iops_max" : 422056,\n'
                                             '        "iops_mean" : '
                                             '420286.428571,\n'
                                             '        "iops_stddev" : '
                                             '934.806662,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 21,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 1.919441,\n'
                                             '        "4" : 98.072977,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 93.588456,\n'
                                             '        "20" : 5.184796,\n'
                                             '        "50" : 1.186323,\n'
                                             '        "100" : 0.035801,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=141818: Wed Oct 26 04:39:28 '
                                             '2022\n'
                                             '  read: IOPS=420k, BW=205MiB/s '
                                             '(215MB/s)(1436MiB/7001msec)\n'
                                             '    slat (nsec): min=210, '
                                             'max=28404, avg=272.23, '
                                             'stdev=44.42\n'
                                             '    clat (nsec): min=360, '
                                             'max=125015, avg=9017.81, '
                                             'stdev=2595.88\n'
                                             '     lat (usec): min=5, max=125, '
                                             'avg= 9.29, stdev= 2.60\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6752],  '
                                             '5.00th=[ 7456], 10.00th=[ 7840], '
                                             '20.00th=[ 7968],\n'
                                             '     | 30.00th=[ 8032], '
                                             '40.00th=[ 8160], 50.00th=[ '
                                             '8256], 60.00th=[ 9152],\n'
                                             '     | 70.00th=[ 9280], '
                                             '80.00th=[ 9408], 90.00th=[ '
                                             '9664], 95.00th=[10560],\n'
                                             '     | 99.00th=[20352], '
                                             '99.50th=[22144], '
                                             '99.90th=[29568], '
                                             '99.95th=[36096],\n'
                                             '     | 99.99th=[69120]\n'
                                             '   bw (  KiB/s): min=209329, '
                                             'max=211028, per=100.00%, '
                                             'avg=210143.43, stdev=467.37, '
                                             'samples=14\n'
                                             '   iops        : min=418658, '
                                             'max=422056, avg=420286.43, '
                                             'stdev=934.81, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=93.59%, 20=5.18%, '
                                             '50=1.19%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=21, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=1.9%, '
                                             '4=98.1%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2941273,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=205MiB/s (215MB/s), '
                                             '205MiB/s-205MiB/s '
                                             '(215MB/s-215MB/s), io=1436MiB '
                                             '(1506MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.output'),
                                   'state': {'begin': 1666744879.414543,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7872753143310547,
                                             'end': 1666744880.2018182,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_219.state')},
                       'cmd_22': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.output'),
                                  'state': {'begin': 1666743937.0274687,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7875986099243164,
                                            'end': 1666743937.8150673,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_22.state')},
                       'cmd_220': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.output'),
                                   'state': {'begin': 1666744882.2170968,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7816946506500244,
                                             'end': 1666744882.9987915,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_220.state')},
                       'cmd_221': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.output'),
                                   'state': {'begin': 1666744883.0009136,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.48667311668396,
                                             'end': 1666744894.4875867,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_221.state')},
                       'cmd_222': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751983,\n'
                                             '  "timestamp_ms" : '
                                             '1666751983603,\n'
                                             '  "time" : "Wed Oct 26 04:39:43 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1417665024,\n'
                                             '        "io_kbytes" : 1384438,\n'
                                             '        "bw_bytes" : 202494647,\n'
                                             '        "bw" : 197748,\n'
                                             '        "iops" : 395496.929010,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2768874,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 170,\n'
                                             '          "max" : 8436,\n'
                                             '          "mean" : 264.949002,\n'
                                             '          "stddev" : 22.607211,\n'
                                             '          "N" : 2768874\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2355,\n'
                                             '          "max" : 122710,\n'
                                             '          "mean" : 9554.498064,\n'
                                             '          "stddev" : '
                                             '2412.332312,\n'
                                             '          "N" : 2768877,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6432,\n'
                                             '            "5.000000" : 7392,\n'
                                             '            "10.000000" : 8032,\n'
                                             '            "20.000000" : 8768,\n'
                                             '            "30.000000" : 9152,\n'
                                             '            "40.000000" : 9152,\n'
                                             '            "50.000000" : 9280,\n'
                                             '            "60.000000" : 9408,\n'
                                             '            "70.000000" : 9536,\n'
                                             '            "80.000000" : 9792,\n'
                                             '            "90.000000" : '
                                             '10432,\n'
                                             '            "95.000000" : '
                                             '11840,\n'
                                             '            "99.000000" : '
                                             '20096,\n'
                                             '            "99.500000" : '
                                             '22144,\n'
                                             '            "99.900000" : '
                                             '28032,\n'
                                             '            "99.950000" : '
                                             '35072,\n'
                                             '            "99.990000" : 69120\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5390,\n'
                                             '          "max" : 122971,\n'
                                             '          "mean" : 9819.447028,\n'
                                             '          "stddev" : '
                                             '2413.593380,\n'
                                             '          "N" : 2768877\n'
                                             '        },\n'
                                             '        "bw_min" : 197589,\n'
                                             '        "bw_max" : 198348,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '197841.000000,\n'
                                             '        "bw_dev" : 217.461933,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 395178,\n'
                                             '        "iops_max" : 396696,\n'
                                             '        "iops_mean" : '
                                             '395682.071429,\n'
                                             '        "iops_stddev" : '
                                             '435.142499,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 24,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 11.586082,\n'
                                             '        "4" : 88.411318,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 87.145894,\n'
                                             '        "20" : 11.771283,\n'
                                             '        "50" : 1.041795,\n'
                                             '        "100" : 0.036007,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=141869: Wed Oct 26 04:39:43 '
                                             '2022\n'
                                             '  read: IOPS=395k, BW=193MiB/s '
                                             '(202MB/s)(1352MiB/7001msec)\n'
                                             '    slat (nsec): min=170, '
                                             'max=8436, avg=264.95, '
                                             'stdev=22.61\n'
                                             '    clat (usec): min=2, max=122, '
                                             'avg= 9.55, stdev= 2.41\n'
                                             '     lat (usec): min=5, max=122, '
                                             'avg= 9.82, stdev= 2.41\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6432],  '
                                             '5.00th=[ 7392], 10.00th=[ 8032], '
                                             '20.00th=[ 8768],\n'
                                             '     | 30.00th=[ 9152], '
                                             '40.00th=[ 9152], 50.00th=[ '
                                             '9280], 60.00th=[ 9408],\n'
                                             '     | 70.00th=[ 9536], '
                                             '80.00th=[ 9792], '
                                             '90.00th=[10432], '
                                             '95.00th=[11840],\n'
                                             '     | 99.00th=[20096], '
                                             '99.50th=[22144], '
                                             '99.90th=[28032], '
                                             '99.95th=[35072],\n'
                                             '     | 99.99th=[69120]\n'
                                             '   bw (  KiB/s): min=197589, '
                                             'max=198348, per=100.00%, '
                                             'avg=197841.00, stdev=217.46, '
                                             'samples=14\n'
                                             '   iops        : min=395178, '
                                             'max=396696, avg=395682.07, '
                                             'stdev=435.14, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=87.15%, 20=11.77%, 50=1.04%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=24, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=11.6%, 4=88.4%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2768874,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=193MiB/s (202MB/s), '
                                             '193MiB/s-193MiB/s '
                                             '(202MB/s-202MB/s), io=1352MiB '
                                             '(1418MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.output'),
                                   'state': {'begin': 1666744894.4904203,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.791827917098999,
                                             'end': 1666744895.2822483,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_222.state')},
                       'cmd_223': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.output'),
                                   'state': {'begin': 1666744897.3290775,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7823565006256104,
                                             'end': 1666744898.111434,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_223.state')},
                       'cmd_224': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.output'),
                                   'state': {'begin': 1666744898.1135473,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.458956718444824,
                                             'end': 1666744909.572504,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_224.state')},
                       'cmd_225': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666751998,\n'
                                             '  "timestamp_ms" : '
                                             '1666751998687,\n'
                                             '  "time" : "Wed Oct 26 04:39:58 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1425394688,\n'
                                             '        "io_kbytes" : 1391987,\n'
                                             '        "bw_bytes" : 203598727,\n'
                                             '        "bw" : 198826,\n'
                                             '        "iops" : 397653.335238,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2783971,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 180,\n'
                                             '          "max" : 14417,\n'
                                             '          "mean" : 263.192554,\n'
                                             '          "stddev" : 26.016764,\n'
                                             '          "N" : 2783971\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 391,\n'
                                             '          "max" : 156223,\n'
                                             '          "mean" : 9514.911947,\n'
                                             '          "stddev" : '
                                             '2418.859053,\n'
                                             '          "N" : 2783974,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6432,\n'
                                             '            "5.000000" : 7456,\n'
                                             '            "10.000000" : 8160,\n'
                                             '            "20.000000" : 8768,\n'
                                             '            "30.000000" : 9024,\n'
                                             '            "40.000000" : 9152,\n'
                                             '            "50.000000" : 9280,\n'
                                             '            "60.000000" : 9280,\n'
                                             '            "70.000000" : 9408,\n'
                                             '            "80.000000" : 9536,\n'
                                             '            "90.000000" : '
                                             '10176,\n'
                                             '            "95.000000" : '
                                             '11584,\n'
                                             '            "99.000000" : '
                                             '20352,\n'
                                             '            "99.500000" : '
                                             '22144,\n'
                                             '            "99.900000" : '
                                             '27776,\n'
                                             '            "99.950000" : '
                                             '34560,\n'
                                             '            "99.990000" : 69120\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5100,\n'
                                             '          "max" : 156484,\n'
                                             '          "mean" : 9778.104498,\n'
                                             '          "stddev" : '
                                             '2419.649924,\n'
                                             '          "N" : 2783974\n'
                                             '        },\n'
                                             '        "bw_min" : 198533,\n'
                                             '        "bw_max" : 199591,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '198952.285714,\n'
                                             '        "bw_dev" : 317.459247,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 397066,\n'
                                             '        "iops_max" : 399182,\n'
                                             '        "iops_mean" : '
                                             '397904.714286,\n'
                                             '        "iops_stddev" : '
                                             '634.979519,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 16,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 7.956728,\n'
                                             '        "4" : 92.041656,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 88.694997,\n'
                                             '        "20" : 10.182757,\n'
                                             '        "50" : 1.083416,\n'
                                             '        "100" : 0.033872,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=141919: Wed Oct 26 04:39:58 '
                                             '2022\n'
                                             '  read: IOPS=398k, BW=194MiB/s '
                                             '(204MB/s)(1359MiB/7001msec)\n'
                                             '    slat (nsec): min=180, '
                                             'max=14417, avg=263.19, '
                                             'stdev=26.02\n'
                                             '    clat (nsec): min=391, '
                                             'max=156223, avg=9514.91, '
                                             'stdev=2418.86\n'
                                             '     lat (usec): min=5, max=156, '
                                             'avg= 9.78, stdev= 2.42\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6432],  '
                                             '5.00th=[ 7456], 10.00th=[ 8160], '
                                             '20.00th=[ 8768],\n'
                                             '     | 30.00th=[ 9024], '
                                             '40.00th=[ 9152], 50.00th=[ '
                                             '9280], 60.00th=[ 9280],\n'
                                             '     | 70.00th=[ 9408], '
                                             '80.00th=[ 9536], '
                                             '90.00th=[10176], '
                                             '95.00th=[11584],\n'
                                             '     | 99.00th=[20352], '
                                             '99.50th=[22144], '
                                             '99.90th=[27776], '
                                             '99.95th=[34560],\n'
                                             '     | 99.99th=[69120]\n'
                                             '   bw (  KiB/s): min=198533, '
                                             'max=199591, per=100.00%, '
                                             'avg=198952.29, stdev=317.46, '
                                             'samples=14\n'
                                             '   iops        : min=397066, '
                                             'max=399182, avg=397904.71, '
                                             'stdev=634.98, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=88.69%, 20=10.18%, 50=1.08%, '
                                             '100=0.03%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=16, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=8.0%, '
                                             '4=92.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2783971,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=194MiB/s (204MB/s), '
                                             '194MiB/s-194MiB/s '
                                             '(204MB/s-204MB/s), io=1359MiB '
                                             '(1425MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.output'),
                                   'state': {'begin': 1666744909.5753038,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7929620742797852,
                                             'end': 1666744910.3682659,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_225.state')},
                       'cmd_226': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.output'),
                                   'state': {'begin': 1666744911.3604612,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.787114143371582,
                                             'end': 1666744912.1475754,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_226.state')},
                       'cmd_227': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.output'),
                                   'state': {'begin': 1666744912.149671,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.040347814559937,
                                             'end': 1666744923.190019,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_227.state')},
                       'cmd_228': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752012,\n'
                                             '  "timestamp_ms" : '
                                             '1666752012376,\n'
                                             '  "time" : "Wed Oct 26 04:40:12 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1110281216,\n'
                                             '        "io_kbytes" : 1084259,\n'
                                             '        "bw_bytes" : 158588946,\n'
                                             '        "bw" : 154872,\n'
                                             '        "iops" : 309743.465219,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2168514,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1292,\n'
                                             '          "max" : 24085,\n'
                                             '          "mean" : 2504.110085,\n'
                                             '          "stddev" : '
                                             '952.888239,\n'
                                             '          "N" : 2168515\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 80,\n'
                                             '          "max" : 198122,\n'
                                             '          "mean" : '
                                             '10032.524192,\n'
                                             '          "stddev" : '
                                             '2637.425346,\n'
                                             '          "N" : 2168517,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7520,\n'
                                             '            "5.000000" : 8032,\n'
                                             '            "10.000000" : 9280,\n'
                                             '            "20.000000" : 9408,\n'
                                             '            "30.000000" : 9408,\n'
                                             '            "40.000000" : 9408,\n'
                                             '            "50.000000" : 9536,\n'
                                             '            "60.000000" : 9536,\n'
                                             '            "70.000000" : 9664,\n'
                                             '            "80.000000" : '
                                             '10688,\n'
                                             '            "90.000000" : '
                                             '11072,\n'
                                             '            "95.000000" : '
                                             '11328,\n'
                                             '            "99.000000" : '
                                             '23168,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '29312,\n'
                                             '            "99.950000" : '
                                             '34560,\n'
                                             '            "99.990000" : 73216\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5731,\n'
                                             '          "max" : 199454,\n'
                                             '          "mean" : '
                                             '12536.634481,\n'
                                             '          "stddev" : '
                                             '2666.909686,\n'
                                             '          "N" : 2168517\n'
                                             '        },\n'
                                             '        "bw_min" : 154763,\n'
                                             '        "bw_max" : 155474,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '154943.461538,\n'
                                             '        "bw_dev" : 227.947953,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 309526,\n'
                                             '        "iops_max" : 310949,\n'
                                             '        "iops_mean" : '
                                             '309887.000000,\n'
                                             '        "iops_stddev" : '
                                             '456.089903,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.742857,\n'
                                             '      "sys_cpu" : 82.900000,\n'
                                             '      "ctx" : 12590,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.086188,\n'
                                             '        "10" : 78.649711,\n'
                                             '        "20" : 19.549194,\n'
                                             '        "50" : 1.657863,\n'
                                             '        "100" : 0.040166,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 3043416,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 15511,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 15511,\n'
                                             '      "util" : 99.035239\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=141958: Wed '
                                             'Oct 26 04:40:12 2022\n'
                                             '  read: IOPS=310k, BW=151MiB/s '
                                             '(159MB/s)(1059MiB/7001msec)\n'
                                             '    slat (nsec): min=1292, '
                                             'max=24085, avg=2504.11, '
                                             'stdev=952.89\n'
                                             '    clat (nsec): min=80, '
                                             'max=198122, avg=10032.52, '
                                             'stdev=2637.43\n'
                                             '     lat (usec): min=5, max=199, '
                                             'avg=12.54, stdev= 2.67\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7520],  '
                                             '5.00th=[ 8032], 10.00th=[ 9280], '
                                             '20.00th=[ 9408],\n'
                                             '     | 30.00th=[ 9408], '
                                             '40.00th=[ 9408], 50.00th=[ '
                                             '9536], 60.00th=[ 9536],\n'
                                             '     | 70.00th=[ 9664], '
                                             '80.00th=[10688], '
                                             '90.00th=[11072], '
                                             '95.00th=[11328],\n'
                                             '     | 99.00th=[23168], '
                                             '99.50th=[24960], '
                                             '99.90th=[29312], '
                                             '99.95th=[34560],\n'
                                             '     | 99.99th=[73216]\n'
                                             '   bw (  KiB/s): min=154763, '
                                             'max=155474, per=100.00%, '
                                             'avg=154943.46, stdev=227.95, '
                                             'samples=13\n'
                                             '   iops        : min=309526, '
                                             'max=310949, avg=309887.00, '
                                             'stdev=456.09, samples=13\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.09%, 10=78.65%, 20=19.55%, '
                                             '50=1.66%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=14.74%, '
                                             'sys=82.90%, ctx=12590, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2168514,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=151MiB/s (159MB/s), '
                                             '151MiB/s-151MiB/s '
                                             '(159MB/s-159MB/s), io=1059MiB '
                                             '(1110MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=3043416/0, '
                                             'merge=0/0, ticks=15511/0, '
                                             'in_queue=15511, util=99.04%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.output'),
                                   'state': {'begin': 1666744923.1928506,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7912318706512451,
                                             'end': 1666744923.9840825,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_228.state')},
                       'cmd_229': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.output'),
                                   'state': {'begin': 1666744924.9830532,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7857077121734619,
                                             'end': 1666744925.768761,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_229.state')},
                       'cmd_23': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.output'),
                                  'state': {'begin': 1666743937.821821,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.486093521118164,
                                            'end': 1666743949.3079145,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_23.state')},
                       'cmd_230': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.output'),
                                   'state': {'begin': 1666744925.7707062,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.034291982650757,
                                             'end': 1666744936.8049982,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_230.state')},
                       'cmd_231': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752025,\n'
                                             '  "timestamp_ms" : '
                                             '1666752025990,\n'
                                             '  "time" : "Wed Oct 26 04:40:25 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1115557888,\n'
                                             '        "io_kbytes" : 1089412,\n'
                                             '        "bw_bytes" : 159342649,\n'
                                             '        "bw" : 155608,\n'
                                             '        "iops" : 311215.540637,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2178820,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1292,\n'
                                             '          "max" : 28123,\n'
                                             '          "mean" : 2499.686404,\n'
                                             '          "stddev" : '
                                             '889.896990,\n'
                                             '          "N" : 2178821\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 70,\n'
                                             '          "max" : 142838,\n'
                                             '          "mean" : 9952.938270,\n'
                                             '          "stddev" : '
                                             '2613.804392,\n'
                                             '          "N" : 2178823,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7392,\n'
                                             '            "5.000000" : 8032,\n'
                                             '            "10.000000" : 9152,\n'
                                             '            "20.000000" : 9280,\n'
                                             '            "30.000000" : 9408,\n'
                                             '            "40.000000" : 9408,\n'
                                             '            "50.000000" : 9536,\n'
                                             '            "60.000000" : 9536,\n'
                                             '            "70.000000" : 9536,\n'
                                             '            "80.000000" : '
                                             '10432,\n'
                                             '            "90.000000" : '
                                             '11072,\n'
                                             '            "95.000000" : '
                                             '11328,\n'
                                             '            "99.000000" : '
                                             '23168,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '29824,\n'
                                             '            "99.950000" : '
                                             '34560,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5541,\n'
                                             '          "max" : 144180,\n'
                                             '          "mean" : '
                                             '12452.624922,\n'
                                             '          "stddev" : '
                                             '2606.970828,\n'
                                             '          "N" : 2178823\n'
                                             '        },\n'
                                             '        "bw_min" : 155411,\n'
                                             '        "bw_max" : 156036,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '155680.285714,\n'
                                             '        "bw_dev" : 179.364017,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 310822,\n'
                                             '        "iops_max" : 312072,\n'
                                             '        "iops_mean" : '
                                             '311360.571429,\n'
                                             '        "iops_stddev" : '
                                             '358.728034,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.085714,\n'
                                             '      "sys_cpu" : 82.542857,\n'
                                             '      "ctx" : 12650,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010281,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.110702,\n'
                                             '        "10" : 79.515885,\n'
                                             '        "20" : 18.645368,\n'
                                             '        "50" : 1.666728,\n'
                                             '        "100" : 0.042362,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 3132143,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 15922,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 15922,\n'
                                             '      "util" : 99.098564\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=141996: Wed '
                                             'Oct 26 04:40:25 2022\n'
                                             '  read: IOPS=311k, BW=152MiB/s '
                                             '(159MB/s)(1064MiB/7001msec)\n'
                                             '    slat (nsec): min=1292, '
                                             'max=28123, avg=2499.69, '
                                             'stdev=889.90\n'
                                             '    clat (nsec): min=70, '
                                             'max=142838, avg=9952.94, '
                                             'stdev=2613.80\n'
                                             '     lat (usec): min=5, max=144, '
                                             'avg=12.45, stdev= 2.61\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7392],  '
                                             '5.00th=[ 8032], 10.00th=[ 9152], '
                                             '20.00th=[ 9280],\n'
                                             '     | 30.00th=[ 9408], '
                                             '40.00th=[ 9408], 50.00th=[ '
                                             '9536], 60.00th=[ 9536],\n'
                                             '     | 70.00th=[ 9536], '
                                             '80.00th=[10432], '
                                             '90.00th=[11072], '
                                             '95.00th=[11328],\n'
                                             '     | 99.00th=[23168], '
                                             '99.50th=[24960], '
                                             '99.90th=[29824], '
                                             '99.95th=[34560],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=155411, '
                                             'max=156036, per=100.00%, '
                                             'avg=155680.29, stdev=179.36, '
                                             'samples=14\n'
                                             '   iops        : min=310822, '
                                             'max=312072, avg=311360.57, '
                                             'stdev=358.73, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.11%, 10=79.52%, 20=18.65%, '
                                             '50=1.67%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=15.09%, '
                                             'sys=82.54%, ctx=12650, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2178820,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=152MiB/s (159MB/s), '
                                             '152MiB/s-152MiB/s '
                                             '(159MB/s-159MB/s), io=1064MiB '
                                             '(1116MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=3132143/0, '
                                             'merge=0/0, ticks=15922/0, '
                                             'in_queue=15922, util=99.10%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.output'),
                                   'state': {'begin': 1666744936.8076468,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7865753173828125,
                                             'end': 1666744937.594222,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_231.state')},
                       'cmd_232': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.output'),
                                   'state': {'begin': 1666744938.592386,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.79380202293396,
                                             'end': 1666744939.386188,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_232.state')},
                       'cmd_233': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.output'),
                                   'state': {'begin': 1666744939.3881023,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.045184850692749,
                                             'end': 1666744950.4332871,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_233.state')},
                       'cmd_234': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752039,\n'
                                             '  "timestamp_ms" : '
                                             '1666752039618,\n'
                                             '  "time" : "Wed Oct 26 04:40:39 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1114947584,\n'
                                             '        "io_kbytes" : 1088816,\n'
                                             '        "bw_bytes" : 159255475,\n'
                                             '        "bw" : 155522,\n'
                                             '        "iops" : 311045.279246,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2177628,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1282,\n'
                                             '          "max" : 115326,\n'
                                             '          "mean" : 2502.982368,\n'
                                             '          "stddev" : '
                                             '919.655007,\n'
                                             '          "N" : 2177629\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 70,\n'
                                             '          "max" : 153448,\n'
                                             '          "mean" : 9965.891609,\n'
                                             '          "stddev" : '
                                             '2636.066476,\n'
                                             '          "N" : 2177631,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7456,\n'
                                             '            "5.000000" : 8032,\n'
                                             '            "10.000000" : 9152,\n'
                                             '            "20.000000" : 9280,\n'
                                             '            "30.000000" : 9408,\n'
                                             '            "40.000000" : 9408,\n'
                                             '            "50.000000" : 9536,\n'
                                             '            "60.000000" : 9536,\n'
                                             '            "70.000000" : 9536,\n'
                                             '            "80.000000" : 9920,\n'
                                             '            "90.000000" : '
                                             '11072,\n'
                                             '            "95.000000" : '
                                             '11200,\n'
                                             '            "99.000000" : '
                                             '23168,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '29312,\n'
                                             '            "99.950000" : '
                                             '34048,\n'
                                             '            "99.990000" : 73216\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5721,\n'
                                             '          "max" : 154770,\n'
                                             '          "mean" : '
                                             '12468.873560,\n'
                                             '          "stddev" : '
                                             '2639.118189,\n'
                                             '          "N" : 2177631\n'
                                             '        },\n'
                                             '        "bw_min" : 155227,\n'
                                             '        "bw_max" : 156354,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '155585.285714,\n'
                                             '        "bw_dev" : 335.643453,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 310454,\n'
                                             '        "iops_max" : 312710,\n'
                                             '        "iops_mean" : '
                                             '311170.571429,\n'
                                             '        "iops_stddev" : '
                                             '671.748776,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.314286,\n'
                                             '      "sys_cpu" : 82.328571,\n'
                                             '      "ctx" : 12629,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.015154,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.107319,\n'
                                             '        "10" : 79.928757,\n'
                                             '        "20" : 18.224600,\n'
                                             '        "50" : 1.676457,\n'
                                             '        "100" : 0.037380,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 3131615,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 15931,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 15931,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=142034: Wed '
                                             'Oct 26 04:40:39 2022\n'
                                             '  read: IOPS=311k, BW=152MiB/s '
                                             '(159MB/s)(1063MiB/7001msec)\n'
                                             '    slat (nsec): min=1282, '
                                             'max=115326, avg=2502.98, '
                                             'stdev=919.66\n'
                                             '    clat (nsec): min=70, '
                                             'max=153448, avg=9965.89, '
                                             'stdev=2636.07\n'
                                             '     lat (usec): min=5, max=154, '
                                             'avg=12.47, stdev= 2.64\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7456],  '
                                             '5.00th=[ 8032], 10.00th=[ 9152], '
                                             '20.00th=[ 9280],\n'
                                             '     | 30.00th=[ 9408], '
                                             '40.00th=[ 9408], 50.00th=[ '
                                             '9536], 60.00th=[ 9536],\n'
                                             '     | 70.00th=[ 9536], '
                                             '80.00th=[ 9920], '
                                             '90.00th=[11072], '
                                             '95.00th=[11200],\n'
                                             '     | 99.00th=[23168], '
                                             '99.50th=[24960], '
                                             '99.90th=[29312], '
                                             '99.95th=[34048],\n'
                                             '     | 99.99th=[73216]\n'
                                             '   bw (  KiB/s): min=155227, '
                                             'max=156354, per=100.00%, '
                                             'avg=155585.29, stdev=335.64, '
                                             'samples=14\n'
                                             '   iops        : min=310454, '
                                             'max=312710, avg=311170.57, '
                                             'stdev=671.75, samples=14\n'
                                             '  lat (nsec)   : 100=0.02%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.11%, 10=79.93%, 20=18.22%, '
                                             '50=1.68%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=15.31%, '
                                             'sys=82.33%, ctx=12629, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2177628,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=152MiB/s (159MB/s), '
                                             '152MiB/s-152MiB/s '
                                             '(159MB/s-159MB/s), io=1063MiB '
                                             '(1115MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=3131615/0, '
                                             'merge=0/0, ticks=15931/0, '
                                             'in_queue=15931, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.output'),
                                   'state': {'begin': 1666744950.435849,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7989151477813721,
                                             'end': 1666744951.234764,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_234.state')},
                       'cmd_235': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.output'),
                                   'state': {'begin': 1666744953.2707112,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7816410064697266,
                                             'end': 1666744954.0523522,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_235.state')},
                       'cmd_236': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.output'),
                                   'state': {'begin': 1666744954.0543783,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.445031881332397,
                                             'end': 1666744965.4994102,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_236.state')},
                       'cmd_237': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752054,\n'
                                             '  "timestamp_ms" : '
                                             '1666752054612,\n'
                                             '  "time" : "Wed Oct 26 04:40:54 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1012919808,\n'
                                             '        "io_kbytes" : 989179,\n'
                                             '        "bw_bytes" : 144682160,\n'
                                             '        "bw" : 141291,\n'
                                             '        "iops" : 282581.916869,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1978356,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 26279,\n'
                                             '          "mean" : 141.316532,\n'
                                             '          "stddev" : 49.171458,\n'
                                             '          "N" : 1978356\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 11882,\n'
                                             '          "max" : 177744,\n'
                                             '          "mean" : '
                                             '13172.391741,\n'
                                             '          "stddev" : '
                                             '2083.866987,\n'
                                             '          "N" : 1978356,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12224,\n'
                                             '            "5.000000" : 12352,\n'
                                             '            "10.000000" : '
                                             '12352,\n'
                                             '            "20.000000" : '
                                             '12608,\n'
                                             '            "30.000000" : '
                                             '12608,\n'
                                             '            "40.000000" : '
                                             '12736,\n'
                                             '            "50.000000" : '
                                             '12864,\n'
                                             '            "60.000000" : '
                                             '12992,\n'
                                             '            "70.000000" : '
                                             '13120,\n'
                                             '            "80.000000" : '
                                             '13248,\n'
                                             '            "90.000000" : '
                                             '13376,\n'
                                             '            "95.000000" : '
                                             '14016,\n'
                                             '            "99.000000" : '
                                             '22912,\n'
                                             '            "99.500000" : '
                                             '23936,\n'
                                             '            "99.900000" : '
                                             '30592,\n'
                                             '            "99.950000" : '
                                             '46336,\n'
                                             '            "99.990000" : 71168\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 12003,\n'
                                             '          "max" : 177944,\n'
                                             '          "mean" : '
                                             '13313.708273,\n'
                                             '          "stddev" : '
                                             '2088.192140,\n'
                                             '          "N" : 1978356\n'
                                             '        },\n'
                                             '        "bw_min" : 141168,\n'
                                             '        "bw_max" : 141492,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '141330.928571,\n'
                                             '        "bw_dev" : 101.098934,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 282336,\n'
                                             '        "iops_max" : 282984,\n'
                                             '        "iops_mean" : '
                                             '282661.928571,\n'
                                             '        "iops_stddev" : '
                                             '201.869137,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 38.900000,\n'
                                             '      "sys_cpu" : 61.071429,\n'
                                             '      "ctx" : 22,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 25.000000,\n'
                                             '        "2" : 50.000000,\n'
                                             '        "4" : 25.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 98.583622,\n'
                                             '        "50" : 1.369925,\n'
                                             '        "100" : 0.039477,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=142084: Wed '
                                             'Oct 26 04:40:54 2022\n'
                                             '  read: IOPS=283k, BW=138MiB/s '
                                             '(145MB/s)(966MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=26279, avg=141.32, '
                                             'stdev=49.17\n'
                                             '    clat (usec): min=11, '
                                             'max=177, avg=13.17, stdev= 2.08\n'
                                             '     lat (usec): min=12, '
                                             'max=177, avg=13.31, stdev= 2.09\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12224],  '
                                             '5.00th=[12352], 10.00th=[12352], '
                                             '20.00th=[12608],\n'
                                             '     | 30.00th=[12608], '
                                             '40.00th=[12736], '
                                             '50.00th=[12864], '
                                             '60.00th=[12992],\n'
                                             '     | 70.00th=[13120], '
                                             '80.00th=[13248], '
                                             '90.00th=[13376], '
                                             '95.00th=[14016],\n'
                                             '     | 99.00th=[22912], '
                                             '99.50th=[23936], '
                                             '99.90th=[30592], '
                                             '99.95th=[46336],\n'
                                             '     | 99.99th=[71168]\n'
                                             '   bw (  KiB/s): min=141168, '
                                             'max=141492, per=100.00%, '
                                             'avg=141330.93, stdev=101.10, '
                                             'samples=14\n'
                                             '   iops        : min=282336, '
                                             'max=282984, avg=282661.93, '
                                             'stdev=201.87, samples=14\n'
                                             '  lat (usec)   : 20=98.58%, '
                                             '50=1.37%, 100=0.04%, 250=0.01%\n'
                                             '  cpu          : usr=38.90%, '
                                             'sys=61.07%, ctx=22, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=25.0%, '
                                             '2=50.0%, 4=25.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1978356,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=138MiB/s (145MB/s), '
                                             '138MiB/s-138MiB/s '
                                             '(145MB/s-145MB/s), io=966MiB '
                                             '(1013MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.output'),
                                   'state': {'begin': 1666744965.5021877,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7887923717498779,
                                             'end': 1666744966.29098,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_237.state')},
                       'cmd_238': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.output'),
                                   'state': {'begin': 1666744968.3584857,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7827889919281006,
                                             'end': 1666744969.1412747,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_238.state')},
                       'cmd_239': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.output'),
                                   'state': {'begin': 1666744969.1431932,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.46291971206665,
                                             'end': 1666744980.606113,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_239.state')},
                       'cmd_24': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751038,\n'
                                            '  "timestamp_ms" : '
                                            '1666751038438,\n'
                                            '  "time" : "Wed Oct 26 04:23:58 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 438721024,\n'
                                            '        "io_kbytes" : 428438,\n'
                                            '        "bw_bytes" : 62665479,\n'
                                            '        "bw" : 61196,\n'
                                            '        "iops" : 122393.372375,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 856876,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 711,\n'
                                            '          "max" : 19336,\n'
                                            '          "mean" : 922.608556,\n'
                                            '          "stddev" : 65.998960,\n'
                                            '          "N" : 856877\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 731,\n'
                                            '          "max" : 1292413,\n'
                                            '          "mean" : 6966.034914,\n'
                                            '          "stddev" : '
                                            '1782.190180,\n'
                                            '          "N" : 856876,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6368,\n'
                                            '            "5.000000" : 6496,\n'
                                            '            "10.000000" : 6560,\n'
                                            '            "20.000000" : 6624,\n'
                                            '            "30.000000" : 6752,\n'
                                            '            "40.000000" : 6752,\n'
                                            '            "50.000000" : 6816,\n'
                                            '            "60.000000" : 6944,\n'
                                            '            "70.000000" : 6944,\n'
                                            '            "80.000000" : 7008,\n'
                                            '            "90.000000" : 7136,\n'
                                            '            "95.000000" : 7264,\n'
                                            '            "99.000000" : 12864,\n'
                                            '            "99.500000" : 13888,\n'
                                            '            "99.900000" : 16512,\n'
                                            '            "99.950000" : 16768,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6993,\n'
                                            '          "max" : 1293966,\n'
                                            '          "mean" : 7888.643483,\n'
                                            '          "stddev" : '
                                            '1784.824087,\n'
                                            '          "N" : 856876\n'
                                            '        },\n'
                                            '        "bw_min" : 61040,\n'
                                            '        "bw_max" : 61478,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '61222.769231,\n'
                                            '        "bw_dev" : 152.265751,\n'
                                            '        "bw_samples" : 13,\n'
                                            '        "iops_min" : 122080,\n'
                                            '        "iops_max" : 122956,\n'
                                            '        "iops_mean" : '
                                            '122445.769231,\n'
                                            '        "iops_stddev" : '
                                            '304.518241,\n'
                                            '        "iops_samples" : 13\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 99.985714,\n'
                                            '      "sys_cpu" : 0.000000,\n'
                                            '      "ctx" : 21,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.313175,\n'
                                            '        "20" : 1.668036,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013771,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-bdev_xnvme: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=138968: Wed Oct 26 04:23:58 '
                                            '2022\n'
                                            '  read: IOPS=122k, BW=59.8MiB/s '
                                            '(62.7MB/s)(418MiB/7001msec)\n'
                                            '    slat (nsec): min=711, '
                                            'max=19336, avg=922.61, '
                                            'stdev=66.00\n'
                                            '    clat (nsec): min=731, '
                                            'max=1292.4k, avg=6966.03, '
                                            'stdev=1782.19\n'
                                            '     lat (usec): min=6, max=1293, '
                                            'avg= 7.89, stdev= 1.78\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6368],  5.00th=[ '
                                            '6496], 10.00th=[ 6560], 20.00th=[ '
                                            '6624],\n'
                                            '     | 30.00th=[ 6752], 40.00th=[ '
                                            '6752], 50.00th=[ 6816], 60.00th=[ '
                                            '6944],\n'
                                            '     | 70.00th=[ 6944], 80.00th=[ '
                                            '7008], 90.00th=[ 7136], 95.00th=[ '
                                            '7264],\n'
                                            '     | 99.00th=[12864], '
                                            '99.50th=[13888], 99.90th=[16512], '
                                            '99.95th=[16768],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=61040, '
                                            'max=61478, per=100.00%, '
                                            'avg=61222.77, stdev=152.27, '
                                            'samples=13\n'
                                            '   iops        : min=122080, '
                                            'max=122956, avg=122445.77, '
                                            'stdev=304.52, samples=13\n'
                                            '  lat (nsec)   : 750=0.01%\n'
                                            '  lat (usec)   : 10=98.31%, '
                                            '20=1.67%, 50=0.01%, 100=0.01%\n'
                                            '  lat (msec)   : 2=0.01%\n'
                                            '  cpu          : usr=99.99%, '
                                            'sys=0.00%, ctx=21, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=856876,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=59.8MiB/s (62.7MB/s), '
                                            '59.8MiB/s-59.8MiB/s '
                                            '(62.7MB/s-62.7MB/s), io=418MiB '
                                            '(439MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.output'),
                                  'state': {'begin': 1666743949.3106942,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.787369966506958,
                                            'end': 1666743950.0980642,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_24.state')},
                       'cmd_240': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752069,\n'
                                             '  "timestamp_ms" : '
                                             '1666752069718,\n'
                                             '  "time" : "Wed Oct 26 04:41:09 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1013155328,\n'
                                             '        "io_kbytes" : 989409,\n'
                                             '        "bw_bytes" : 144715801,\n'
                                             '        "bw" : 141324,\n'
                                             '        "iops" : 282647.621768,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1978816,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 12503,\n'
                                             '          "mean" : 142.496506,\n'
                                             '          "stddev" : 47.180392,\n'
                                             '          "N" : 1978816\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 11873,\n'
                                             '          "max" : 117370,\n'
                                             '          "mean" : '
                                             '13154.827341,\n'
                                             '          "stddev" : '
                                             '1841.054914,\n'
                                             '          "N" : 1978816,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12224,\n'
                                             '            "5.000000" : 12352,\n'
                                             '            "10.000000" : '
                                             '12352,\n'
                                             '            "20.000000" : '
                                             '12608,\n'
                                             '            "30.000000" : '
                                             '12608,\n'
                                             '            "40.000000" : '
                                             '12736,\n'
                                             '            "50.000000" : '
                                             '12864,\n'
                                             '            "60.000000" : '
                                             '12992,\n'
                                             '            "70.000000" : '
                                             '13120,\n'
                                             '            "80.000000" : '
                                             '13248,\n'
                                             '            "90.000000" : '
                                             '13376,\n'
                                             '            "95.000000" : '
                                             '13888,\n'
                                             '            "99.000000" : '
                                             '22912,\n'
                                             '            "99.500000" : '
                                             '23936,\n'
                                             '            "99.900000" : '
                                             '29568,\n'
                                             '            "99.950000" : '
                                             '35072,\n'
                                             '            "99.990000" : 66048\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11993,\n'
                                             '          "max" : 117570,\n'
                                             '          "mean" : '
                                             '13297.323847,\n'
                                             '          "stddev" : '
                                             '1846.116285,\n'
                                             '          "N" : 1978816\n'
                                             '        },\n'
                                             '        "bw_min" : 141129,\n'
                                             '        "bw_max" : 141560,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '141360.928571,\n'
                                             '        "bw_dev" : 153.096483,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 282256,\n'
                                             '        "iops_max" : 283120,\n'
                                             '        "iops_mean" : '
                                             '282722.000000,\n'
                                             '        "iops_stddev" : '
                                             '306.117122,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 41.242857,\n'
                                             '      "sys_cpu" : 58.728571,\n'
                                             '      "ctx" : 20,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 25.000000,\n'
                                             '        "2" : 50.000000,\n'
                                             '        "4" : 25.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 98.730200,\n'
                                             '        "50" : 1.228361,\n'
                                             '        "100" : 0.041237,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=142133: Wed '
                                             'Oct 26 04:41:09 2022\n'
                                             '  read: IOPS=283k, BW=138MiB/s '
                                             '(145MB/s)(966MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=12503, avg=142.50, '
                                             'stdev=47.18\n'
                                             '    clat (usec): min=11, '
                                             'max=117, avg=13.15, stdev= 1.84\n'
                                             '     lat (usec): min=11, '
                                             'max=117, avg=13.30, stdev= 1.85\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12224],  '
                                             '5.00th=[12352], 10.00th=[12352], '
                                             '20.00th=[12608],\n'
                                             '     | 30.00th=[12608], '
                                             '40.00th=[12736], '
                                             '50.00th=[12864], '
                                             '60.00th=[12992],\n'
                                             '     | 70.00th=[13120], '
                                             '80.00th=[13248], '
                                             '90.00th=[13376], '
                                             '95.00th=[13888],\n'
                                             '     | 99.00th=[22912], '
                                             '99.50th=[23936], '
                                             '99.90th=[29568], '
                                             '99.95th=[35072],\n'
                                             '     | 99.99th=[66048]\n'
                                             '   bw (  KiB/s): min=141129, '
                                             'max=141560, per=100.00%, '
                                             'avg=141360.93, stdev=153.10, '
                                             'samples=14\n'
                                             '   iops        : min=282256, '
                                             'max=283120, avg=282722.00, '
                                             'stdev=306.12, samples=14\n'
                                             '  lat (usec)   : 20=98.73%, '
                                             '50=1.23%, 100=0.04%, 250=0.01%\n'
                                             '  cpu          : usr=41.24%, '
                                             'sys=58.73%, ctx=20, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=25.0%, '
                                             '2=50.0%, 4=25.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1978816,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=138MiB/s (145MB/s), '
                                             '138MiB/s-138MiB/s '
                                             '(145MB/s-145MB/s), io=966MiB '
                                             '(1013MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.output'),
                                   'state': {'begin': 1666744980.6090484,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7900142669677734,
                                             'end': 1666744981.3990626,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_240.state')},
                       'cmd_241': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.output'),
                                   'state': {'begin': 1666744983.4507341,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7813313007354736,
                                             'end': 1666744984.2320654,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_241.state')},
                       'cmd_242': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.output'),
                                   'state': {'begin': 1666744984.234375,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.442209720611572,
                                             'end': 1666744995.6765847,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_242.state')},
                       'cmd_243': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752084,\n'
                                             '  "timestamp_ms" : '
                                             '1666752084789,\n'
                                             '  "time" : "Wed Oct 26 04:41:24 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1017103872,\n'
                                             '        "io_kbytes" : 993265,\n'
                                             '        "bw_bytes" : 145279798,\n'
                                             '        "bw" : 141874,\n'
                                             '        "iops" : 283749.178689,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1986528,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 13124,\n'
                                             '          "mean" : 141.327949,\n'
                                             '          "stddev" : 46.350036,\n'
                                             '          "N" : 1986528\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 11853,\n'
                                             '          "max" : 118873,\n'
                                             '          "mean" : '
                                             '13131.754659,\n'
                                             '          "stddev" : '
                                             '1847.869899,\n'
                                             '          "N" : 1986528,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12224,\n'
                                             '            "5.000000" : 12352,\n'
                                             '            "10.000000" : '
                                             '12352,\n'
                                             '            "20.000000" : '
                                             '12480,\n'
                                             '            "30.000000" : '
                                             '12608,\n'
                                             '            "40.000000" : '
                                             '12736,\n'
                                             '            "50.000000" : '
                                             '12864,\n'
                                             '            "60.000000" : '
                                             '12992,\n'
                                             '            "70.000000" : '
                                             '12992,\n'
                                             '            "80.000000" : '
                                             '13120,\n'
                                             '            "90.000000" : '
                                             '13376,\n'
                                             '            "95.000000" : '
                                             '13888,\n'
                                             '            "99.000000" : '
                                             '22912,\n'
                                             '            "99.500000" : '
                                             '23936,\n'
                                             '            "99.900000" : '
                                             '29568,\n'
                                             '            "99.950000" : '
                                             '35072,\n'
                                             '            "99.990000" : 66048\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11963,\n'
                                             '          "max" : 119083,\n'
                                             '          "mean" : '
                                             '13273.082608,\n'
                                             '          "stddev" : '
                                             '1853.093830,\n'
                                             '          "N" : 1986528\n'
                                             '        },\n'
                                             '        "bw_min" : 141696,\n'
                                             '        "bw_max" : 142276,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '141915.000000,\n'
                                             '        "bw_dev" : 201.676817,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 283392,\n'
                                             '        "iops_max" : 284552,\n'
                                             '        "iops_mean" : '
                                             '283829.928571,\n'
                                             '        "iops_stddev" : '
                                             '403.275906,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 38.557143,\n'
                                             '      "sys_cpu" : 61.414286,\n'
                                             '      "ctx" : 23,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 25.000000,\n'
                                             '        "2" : 50.000000,\n'
                                             '        "4" : 25.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 98.732412,\n'
                                             '        "50" : 1.227670,\n'
                                             '        "100" : 0.039718,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=142182: Wed '
                                             'Oct 26 04:41:24 2022\n'
                                             '  read: IOPS=284k, BW=139MiB/s '
                                             '(145MB/s)(970MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=13124, avg=141.33, '
                                             'stdev=46.35\n'
                                             '    clat (usec): min=11, '
                                             'max=118, avg=13.13, stdev= 1.85\n'
                                             '     lat (usec): min=11, '
                                             'max=119, avg=13.27, stdev= 1.85\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12224],  '
                                             '5.00th=[12352], 10.00th=[12352], '
                                             '20.00th=[12480],\n'
                                             '     | 30.00th=[12608], '
                                             '40.00th=[12736], '
                                             '50.00th=[12864], '
                                             '60.00th=[12992],\n'
                                             '     | 70.00th=[12992], '
                                             '80.00th=[13120], '
                                             '90.00th=[13376], '
                                             '95.00th=[13888],\n'
                                             '     | 99.00th=[22912], '
                                             '99.50th=[23936], '
                                             '99.90th=[29568], '
                                             '99.95th=[35072],\n'
                                             '     | 99.99th=[66048]\n'
                                             '   bw (  KiB/s): min=141696, '
                                             'max=142276, per=100.00%, '
                                             'avg=141915.00, stdev=201.68, '
                                             'samples=14\n'
                                             '   iops        : min=283392, '
                                             'max=284552, avg=283829.93, '
                                             'stdev=403.28, samples=14\n'
                                             '  lat (usec)   : 20=98.73%, '
                                             '50=1.23%, 100=0.04%, 250=0.01%\n'
                                             '  cpu          : usr=38.56%, '
                                             'sys=61.41%, ctx=23, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=25.0%, '
                                             '2=50.0%, 4=25.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1986528,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=139MiB/s (145MB/s), '
                                             '139MiB/s-139MiB/s '
                                             '(145MB/s-145MB/s), io=970MiB '
                                             '(1017MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.output'),
                                   'state': {'begin': 1666744995.6792803,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7905035018920898,
                                             'end': 1666744996.4697838,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_243.state')},
                       'cmd_244': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.output'),
                                   'state': {'begin': 1666744998.5398793,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7882885932922363,
                                             'end': 1666744999.328168,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_244.state')},
                       'cmd_245': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.output'),
                                   'state': {'begin': 1666744999.3303547,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.473461866378784,
                                             'end': 1666745010.8038166,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_245.state')},
                       'cmd_246': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752099,\n'
                                             '  "timestamp_ms" : '
                                             '1666752099916,\n'
                                             '  "time" : "Wed Oct 26 04:41:39 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1077817344,\n'
                                             '        "io_kbytes" : 1052556,\n'
                                             '        "bw_bytes" : 153951913,\n'
                                             '        "bw" : 150343,\n'
                                             '        "iops" : 300686.759034,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2105108,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1532,\n'
                                             '          "max" : 20038,\n'
                                             '          "mean" : 2804.261997,\n'
                                             '          "stddev" : '
                                             '1140.383516,\n'
                                             '          "N" : 2105109\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 190,\n'
                                             '          "max" : 124213,\n'
                                             '          "mean" : '
                                             '10260.727332,\n'
                                             '          "stddev" : '
                                             '2255.624306,\n'
                                             '          "N" : 2105111,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 8512,\n'
                                             '            "5.000000" : 8640,\n'
                                             '            "10.000000" : 8896,\n'
                                             '            "20.000000" : 9792,\n'
                                             '            "30.000000" : 9792,\n'
                                             '            "40.000000" : 9920,\n'
                                             '            "50.000000" : 9920,\n'
                                             '            "60.000000" : 9920,\n'
                                             '            "70.000000" : '
                                             '10048,\n'
                                             '            "80.000000" : '
                                             '10176,\n'
                                             '            "90.000000" : '
                                             '11200,\n'
                                             '            "95.000000" : '
                                             '11456,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '23424,\n'
                                             '            "99.900000" : '
                                             '29824,\n'
                                             '            "99.950000" : '
                                             '38144,\n'
                                             '            "99.990000" : 68096\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5651,\n'
                                             '          "max" : 125786,\n'
                                             '          "mean" : '
                                             '13064.989316,\n'
                                             '          "stddev" : '
                                             '2336.079254,\n'
                                             '          "N" : 2105111\n'
                                             '        },\n'
                                             '        "bw_min" : 150030,\n'
                                             '        "bw_max" : 150660,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '150378.142857,\n'
                                             '        "bw_dev" : 229.647508,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 300060,\n'
                                             '        "iops_max" : 301320,\n'
                                             '        "iops_mean" : '
                                             '300756.285714,\n'
                                             '        "iops_stddev" : '
                                             '459.295016,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.271429,\n'
                                             '      "sys_cpu" : 67.700000,\n'
                                             '      "ctx" : 21,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.013824,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.021139,\n'
                                             '        "10" : 66.454738,\n'
                                             '        "20" : 32.398053,\n'
                                             '        "50" : 1.065931,\n'
                                             '        "100" : 0.044938,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142231: Wed '
                                             'Oct 26 04:41:39 2022\n'
                                             '  read: IOPS=301k, BW=147MiB/s '
                                             '(154MB/s)(1028MiB/7001msec)\n'
                                             '    slat (nsec): min=1532, '
                                             'max=20038, avg=2804.26, '
                                             'stdev=1140.38\n'
                                             '    clat (nsec): min=190, '
                                             'max=124213, avg=10260.73, '
                                             'stdev=2255.62\n'
                                             '     lat (usec): min=5, max=125, '
                                             'avg=13.06, stdev= 2.34\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 8512],  '
                                             '5.00th=[ 8640], 10.00th=[ 8896], '
                                             '20.00th=[ 9792],\n'
                                             '     | 30.00th=[ 9792], '
                                             '40.00th=[ 9920], 50.00th=[ '
                                             '9920], 60.00th=[ 9920],\n'
                                             '     | 70.00th=[10048], '
                                             '80.00th=[10176], '
                                             '90.00th=[11200], '
                                             '95.00th=[11456],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[23424], '
                                             '99.90th=[29824], '
                                             '99.95th=[38144],\n'
                                             '     | 99.99th=[68096]\n'
                                             '   bw (  KiB/s): min=150030, '
                                             'max=150660, per=100.00%, '
                                             'avg=150378.14, stdev=229.65, '
                                             'samples=14\n'
                                             '   iops        : min=300060, '
                                             'max=301320, avg=300756.29, '
                                             'stdev=459.30, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.02%, '
                                             '10=66.45%, 20=32.40%, 50=1.07%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=32.27%, '
                                             'sys=67.70%, ctx=21, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2105108,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=147MiB/s (154MB/s), '
                                             '147MiB/s-147MiB/s '
                                             '(154MB/s-154MB/s), io=1028MiB '
                                             '(1078MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.output'),
                                   'state': {'begin': 1666745010.8066304,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.9443705081939697,
                                             'end': 1666745011.751001,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_246.state')},
                       'cmd_247': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.output'),
                                   'state': {'begin': 1666745013.8148746,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7813012599945068,
                                             'end': 1666745014.596176,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_247.state')},
                       'cmd_248': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.output'),
                                   'state': {'begin': 1666745014.5983183,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.47979736328125,
                                             'end': 1666745026.0781157,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_248.state')},
                       'cmd_249': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752115,\n'
                                             '  "timestamp_ms" : '
                                             '1666752115189,\n'
                                             '  "time" : "Wed Oct 26 04:41:55 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1060443136,\n'
                                             '        "io_kbytes" : 1035589,\n'
                                             '        "bw_bytes" : 151470237,\n'
                                             '        "bw" : 147920,\n'
                                             '        "iops" : 295839.737180,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2071174,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1542,\n'
                                             '          "max" : 28865,\n'
                                             '          "mean" : 2851.573909,\n'
                                             '          "stddev" : '
                                             '1251.286691,\n'
                                             '          "N" : 2071175\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 190,\n'
                                             '          "max" : 120766,\n'
                                             '          "mean" : '
                                             '10416.253545,\n'
                                             '          "stddev" : '
                                             '2328.765258,\n'
                                             '          "N" : 2071177,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 8640,\n'
                                             '            "5.000000" : 8768,\n'
                                             '            "10.000000" : 8768,\n'
                                             '            "20.000000" : 8896,\n'
                                             '            "30.000000" : 9024,\n'
                                             '            "40.000000" : 9920,\n'
                                             '            "50.000000" : '
                                             '10048,\n'
                                             '            "60.000000" : '
                                             '10176,\n'
                                             '            "70.000000" : '
                                             '11072,\n'
                                             '            "80.000000" : '
                                             '11328,\n'
                                             '            "90.000000" : '
                                             '11456,\n'
                                             '            "95.000000" : '
                                             '11584,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '23424,\n'
                                             '            "99.900000" : '
                                             '29824,\n'
                                             '            "99.950000" : '
                                             '34048,\n'
                                             '            "99.990000" : 66048\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5650,\n'
                                             '          "max" : 122359,\n'
                                             '          "mean" : '
                                             '13267.826877,\n'
                                             '          "stddev" : '
                                             '2220.362076,\n'
                                             '          "N" : 2071177\n'
                                             '        },\n'
                                             '        "bw_min" : 147590,\n'
                                             '        "bw_max" : 148421,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '147953.571429,\n'
                                             '        "bw_dev" : 238.447451,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 295180,\n'
                                             '        "iops_max" : 296842,\n'
                                             '        "iops_mean" : '
                                             '295907.071429,\n'
                                             '        "iops_stddev" : '
                                             '476.984513,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 30.214286,\n'
                                             '      "sys_cpu" : 69.757143,\n'
                                             '      "ctx" : 22,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.016512,\n'
                                             '        "10" : 42.578412,\n'
                                             '        "20" : 56.246361,\n'
                                             '        "50" : 1.119945,\n'
                                             '        "100" : 0.037129,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142280: Wed '
                                             'Oct 26 04:41:55 2022\n'
                                             '  read: IOPS=296k, BW=144MiB/s '
                                             '(151MB/s)(1011MiB/7001msec)\n'
                                             '    slat (nsec): min=1542, '
                                             'max=28865, avg=2851.57, '
                                             'stdev=1251.29\n'
                                             '    clat (nsec): min=190, '
                                             'max=120766, avg=10416.25, '
                                             'stdev=2328.77\n'
                                             '     lat (usec): min=5, max=122, '
                                             'avg=13.27, stdev= 2.22\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 8640],  '
                                             '5.00th=[ 8768], 10.00th=[ 8768], '
                                             '20.00th=[ 8896],\n'
                                             '     | 30.00th=[ 9024], '
                                             '40.00th=[ 9920], '
                                             '50.00th=[10048], '
                                             '60.00th=[10176],\n'
                                             '     | 70.00th=[11072], '
                                             '80.00th=[11328], '
                                             '90.00th=[11456], '
                                             '95.00th=[11584],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[23424], '
                                             '99.90th=[29824], '
                                             '99.95th=[34048],\n'
                                             '     | 99.99th=[66048]\n'
                                             '   bw (  KiB/s): min=147590, '
                                             'max=148421, per=100.00%, '
                                             'avg=147953.57, stdev=238.45, '
                                             'samples=14\n'
                                             '   iops        : min=295180, '
                                             'max=296842, avg=295907.07, '
                                             'stdev=476.98, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.02%, '
                                             '10=42.58%, 20=56.25%, 50=1.12%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=30.21%, '
                                             'sys=69.76%, ctx=22, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2071174,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=144MiB/s (151MB/s), '
                                             '144MiB/s-144MiB/s '
                                             '(151MB/s-151MB/s), io=1011MiB '
                                             '(1060MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.output'),
                                   'state': {'begin': 1666745026.081021,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7926065921783447,
                                             'end': 1666745026.8736277,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_249.state')},
                       'cmd_25': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.output'),
                                  'state': {'begin': 1666743952.1315753,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7893722057342529,
                                            'end': 1666743952.9209476,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_25.state')},
                       'cmd_250': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.output'),
                                   'state': {'begin': 1666745028.9053721,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7826261520385742,
                                             'end': 1666745029.6879983,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_250.state')},
                       'cmd_251': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.output'),
                                   'state': {'begin': 1666745029.6903412,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.451205968856812,
                                             'end': 1666745041.1415472,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_251.state')},
                       'cmd_252': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752130,\n'
                                             '  "timestamp_ms" : '
                                             '1666752130258,\n'
                                             '  "time" : "Wed Oct 26 04:42:10 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1063592960,\n'
                                             '        "io_kbytes" : 1038665,\n'
                                             '        "bw_bytes" : 151920148,\n'
                                             '        "bw" : 148359,\n'
                                             '        "iops" : 296718.468790,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2077326,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1533,\n'
                                             '          "max" : 57908,\n'
                                             '          "mean" : 2828.639420,\n'
                                             '          "stddev" : '
                                             '1225.631029,\n'
                                             '          "N" : 2077327\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 210,\n'
                                             '          "max" : 121017,\n'
                                             '          "mean" : '
                                             '10398.118086,\n'
                                             '          "stddev" : '
                                             '2330.860370,\n'
                                             '          "N" : 2077329,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 8640,\n'
                                             '            "5.000000" : 8768,\n'
                                             '            "10.000000" : 8768,\n'
                                             '            "20.000000" : 8896,\n'
                                             '            "30.000000" : 9920,\n'
                                             '            "40.000000" : 9920,\n'
                                             '            "50.000000" : '
                                             '10048,\n'
                                             '            "60.000000" : '
                                             '10176,\n'
                                             '            "70.000000" : '
                                             '10944,\n'
                                             '            "80.000000" : '
                                             '11200,\n'
                                             '            "90.000000" : '
                                             '11328,\n'
                                             '            "95.000000" : '
                                             '11584,\n'
                                             '            "99.000000" : '
                                             '20864,\n'
                                             '            "99.500000" : '
                                             '23424,\n'
                                             '            "99.900000" : '
                                             '30080,\n'
                                             '            "99.950000" : '
                                             '35072,\n'
                                             '            "99.990000" : 67072\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5670,\n'
                                             '          "max" : 122710,\n'
                                             '          "mean" : '
                                             '13226.758125,\n'
                                             '          "stddev" : '
                                             '2280.945948,\n'
                                             '          "N" : 2077329\n'
                                             '        },\n'
                                             '        "bw_min" : 147572,\n'
                                             '        "bw_max" : 148824,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '148401.285714,\n'
                                             '        "bw_dev" : 298.527522,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 295144,\n'
                                             '        "iops_max" : 297649,\n'
                                             '        "iops_mean" : '
                                             '296802.642857,\n'
                                             '        "iops_stddev" : '
                                             '597.164017,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.328571,\n'
                                             '      "sys_cpu" : 67.642857,\n'
                                             '      "ctx" : 29,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.023925,\n'
                                             '        "10" : 44.046240,\n'
                                             '        "20" : 54.765116,\n'
                                             '        "50" : 1.121779,\n'
                                             '        "100" : 0.039426,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142330: Wed '
                                             'Oct 26 04:42:10 2022\n'
                                             '  read: IOPS=297k, BW=145MiB/s '
                                             '(152MB/s)(1014MiB/7001msec)\n'
                                             '    slat (nsec): min=1533, '
                                             'max=57908, avg=2828.64, '
                                             'stdev=1225.63\n'
                                             '    clat (nsec): min=210, '
                                             'max=121017, avg=10398.12, '
                                             'stdev=2330.86\n'
                                             '     lat (usec): min=5, max=122, '
                                             'avg=13.23, stdev= 2.28\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 8640],  '
                                             '5.00th=[ 8768], 10.00th=[ 8768], '
                                             '20.00th=[ 8896],\n'
                                             '     | 30.00th=[ 9920], '
                                             '40.00th=[ 9920], '
                                             '50.00th=[10048], '
                                             '60.00th=[10176],\n'
                                             '     | 70.00th=[10944], '
                                             '80.00th=[11200], '
                                             '90.00th=[11328], '
                                             '95.00th=[11584],\n'
                                             '     | 99.00th=[20864], '
                                             '99.50th=[23424], '
                                             '99.90th=[30080], '
                                             '99.95th=[35072],\n'
                                             '     | 99.99th=[67072]\n'
                                             '   bw (  KiB/s): min=147572, '
                                             'max=148824, per=100.00%, '
                                             'avg=148401.29, stdev=298.53, '
                                             'samples=14\n'
                                             '   iops        : min=295144, '
                                             'max=297649, avg=296802.64, '
                                             'stdev=597.16, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.02%, '
                                             '10=44.05%, 20=54.77%, 50=1.12%, '
                                             '100=0.04%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=32.33%, '
                                             'sys=67.64%, ctx=29, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2077326,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=145MiB/s (152MB/s), '
                                             '145MiB/s-145MiB/s '
                                             '(152MB/s-152MB/s), io=1014MiB '
                                             '(1064MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.output'),
                                   'state': {'begin': 1666745041.1447248,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7829673290252686,
                                             'end': 1666745041.9276922,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_252.state')},
                       'cmd_253': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.output'),
                                   'state': {'begin': 1666745042.9159136,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7921304702758789,
                                             'end': 1666745043.708044,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_253.state')},
                       'cmd_254': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.output'),
                                   'state': {'begin': 1666745043.71043,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.055681467056274,
                                             'end': 1666745054.7661114,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_254.state')},
                       'cmd_255': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752143,\n'
                                             '  "timestamp_ms" : '
                                             '1666752143949,\n'
                                             '  "time" : "Wed Oct 26 04:42:23 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1086656512,\n'
                                             '        "io_kbytes" : 1061188,\n'
                                             '        "bw_bytes" : 155214471,\n'
                                             '        "bw" : 151576,\n'
                                             '        "iops" : 303152.692473,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2122372,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1372,\n'
                                             '          "max" : 112582,\n'
                                             '          "mean" : 2810.550236,\n'
                                             '          "stddev" : '
                                             '1247.731970,\n'
                                             '          "N" : 2122373\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 125957,\n'
                                             '          "mean" : '
                                             '10142.478753,\n'
                                             '          "stddev" : '
                                             '2569.151273,\n'
                                             '          "N" : 2122375,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6880,\n'
                                             '            "5.000000" : 7968,\n'
                                             '            "10.000000" : 8256,\n'
                                             '            "20.000000" : 9536,\n'
                                             '            "30.000000" : 9664,\n'
                                             '            "40.000000" : 9664,\n'
                                             '            "50.000000" : 9792,\n'
                                             '            "60.000000" : 9792,\n'
                                             '            "70.000000" : 9792,\n'
                                             '            "80.000000" : '
                                             '10048,\n'
                                             '            "90.000000" : '
                                             '11456,\n'
                                             '            "95.000000" : '
                                             '11968,\n'
                                             '            "99.000000" : '
                                             '23424,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '29824,\n'
                                             '            "99.950000" : '
                                             '34048,\n'
                                             '            "99.990000" : 70144\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5921,\n'
                                             '          "max" : 127369,\n'
                                             '          "mean" : '
                                             '12953.029064,\n'
                                             '          "stddev" : '
                                             '2798.630057,\n'
                                             '          "N" : 2122375\n'
                                             '        },\n'
                                             '        "bw_min" : 151380,\n'
                                             '        "bw_max" : 151845,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '151616.571429,\n'
                                             '        "bw_dev" : 148.988236,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 302760,\n'
                                             '        "iops_max" : 303691,\n'
                                             '        "iops_mean" : '
                                             '303233.214286,\n'
                                             '        "iops_stddev" : '
                                             '298.105861,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 18.328571,\n'
                                             '      "sys_cpu" : 79.457143,\n'
                                             '      "ctx" : 12395,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.024925,\n'
                                             '        "4" : 0.110160,\n'
                                             '        "10" : 80.312170,\n'
                                             '        "20" : 17.752307,\n'
                                             '        "50" : 1.758127,\n'
                                             '        "100" : 0.039578,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142369: Wed '
                                             'Oct 26 04:42:23 2022\n'
                                             '  read: IOPS=303k, BW=148MiB/s '
                                             '(155MB/s)(1036MiB/7001msec)\n'
                                             '    slat (nsec): min=1372, '
                                             'max=112582, avg=2810.55, '
                                             'stdev=1247.73\n'
                                             '    clat (nsec): min=110, '
                                             'max=125957, avg=10142.48, '
                                             'stdev=2569.15\n'
                                             '     lat (usec): min=5, max=127, '
                                             'avg=12.95, stdev= 2.80\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6880],  '
                                             '5.00th=[ 7968], 10.00th=[ 8256], '
                                             '20.00th=[ 9536],\n'
                                             '     | 30.00th=[ 9664], '
                                             '40.00th=[ 9664], 50.00th=[ '
                                             '9792], 60.00th=[ 9792],\n'
                                             '     | 70.00th=[ 9792], '
                                             '80.00th=[10048], '
                                             '90.00th=[11456], '
                                             '95.00th=[11968],\n'
                                             '     | 99.00th=[23424], '
                                             '99.50th=[24960], '
                                             '99.90th=[29824], '
                                             '99.95th=[34048],\n'
                                             '     | 99.99th=[70144]\n'
                                             '   bw (  KiB/s): min=151380, '
                                             'max=151845, per=100.00%, '
                                             'avg=151616.57, stdev=148.99, '
                                             'samples=14\n'
                                             '   iops        : min=302760, '
                                             'max=303691, avg=303233.21, '
                                             'stdev=298.11, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.02%, '
                                             '4=0.11%, 10=80.31%, 20=17.75%, '
                                             '50=1.76%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=18.33%, '
                                             'sys=79.46%, ctx=12395, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2122372,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=148MiB/s (155MB/s), '
                                             '148MiB/s-148MiB/s '
                                             '(155MB/s-155MB/s), io=1036MiB '
                                             '(1087MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.output'),
                                   'state': {'begin': 1666745054.7728965,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8778285980224609,
                                             'end': 1666745055.6507251,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_255.state')},
                       'cmd_256': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.output'),
                                   'state': {'begin': 1666745056.7810028,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8336989879608154,
                                             'end': 1666745057.6147017,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_256.state')},
                       'cmd_257': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.output'),
                                   'state': {'begin': 1666745057.6171892,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.133683919906616,
                                             'end': 1666745068.750873,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_257.state')},
                       'cmd_258': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752157,\n'
                                             '  "timestamp_ms" : '
                                             '1666752157934,\n'
                                             '  "time" : "Wed Oct 26 04:42:37 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1083688448,\n'
                                             '        "io_kbytes" : 1058289,\n'
                                             '        "bw_bytes" : 154790522,\n'
                                             '        "bw" : 151162,\n'
                                             '        "iops" : 302324.667905,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2116575,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1362,\n'
                                             '          "max" : 21871,\n'
                                             '          "mean" : 2810.793388,\n'
                                             '          "stddev" : '
                                             '1238.113474,\n'
                                             '          "N" : 2116576\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 126127,\n'
                                             '          "mean" : '
                                             '10178.077711,\n'
                                             '          "stddev" : '
                                             '2575.128921,\n'
                                             '          "N" : 2116578,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6880,\n'
                                             '            "5.000000" : 7968,\n'
                                             '            "10.000000" : 8512,\n'
                                             '            "20.000000" : 9536,\n'
                                             '            "30.000000" : 9664,\n'
                                             '            "40.000000" : 9664,\n'
                                             '            "50.000000" : 9792,\n'
                                             '            "60.000000" : 9792,\n'
                                             '            "70.000000" : 9920,\n'
                                             '            "80.000000" : '
                                             '10048,\n'
                                             '            "90.000000" : '
                                             '11456,\n'
                                             '            "95.000000" : '
                                             '11968,\n'
                                             '            "99.000000" : '
                                             '23424,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '30336,\n'
                                             '            "99.950000" : '
                                             '34560,\n'
                                             '            "99.990000" : 71168\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5841,\n'
                                             '          "max" : 127520,\n'
                                             '          "mean" : '
                                             '12988.870477,\n'
                                             '          "stddev" : '
                                             '2788.918764,\n'
                                             '          "N" : 2116578\n'
                                             '        },\n'
                                             '        "bw_min" : 150921,\n'
                                             '        "bw_max" : 151477,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '151202.642857,\n'
                                             '        "bw_dev" : 172.050801,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 301842,\n'
                                             '        "iops_max" : 302954,\n'
                                             '        "iops_mean" : '
                                             '302405.500000,\n'
                                             '        "iops_stddev" : '
                                             '344.057408,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.585714,\n'
                                             '      "sys_cpu" : 80.171429,\n'
                                             '      "ctx" : 12447,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.019702,\n'
                                             '        "4" : 0.111123,\n'
                                             '        "10" : 79.619905,\n'
                                             '        "20" : 18.496203,\n'
                                             '        "50" : 1.703979,\n'
                                             '        "100" : 0.044884,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142408: Wed '
                                             'Oct 26 04:42:37 2022\n'
                                             '  read: IOPS=302k, BW=148MiB/s '
                                             '(155MB/s)(1033MiB/7001msec)\n'
                                             '    slat (nsec): min=1362, '
                                             'max=21871, avg=2810.79, '
                                             'stdev=1238.11\n'
                                             '    clat (nsec): min=110, '
                                             'max=126127, avg=10178.08, '
                                             'stdev=2575.13\n'
                                             '     lat (usec): min=5, max=127, '
                                             'avg=12.99, stdev= 2.79\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6880],  '
                                             '5.00th=[ 7968], 10.00th=[ 8512], '
                                             '20.00th=[ 9536],\n'
                                             '     | 30.00th=[ 9664], '
                                             '40.00th=[ 9664], 50.00th=[ '
                                             '9792], 60.00th=[ 9792],\n'
                                             '     | 70.00th=[ 9920], '
                                             '80.00th=[10048], '
                                             '90.00th=[11456], '
                                             '95.00th=[11968],\n'
                                             '     | 99.00th=[23424], '
                                             '99.50th=[24960], '
                                             '99.90th=[30336], '
                                             '99.95th=[34560],\n'
                                             '     | 99.99th=[71168]\n'
                                             '   bw (  KiB/s): min=150921, '
                                             'max=151477, per=100.00%, '
                                             'avg=151202.64, stdev=172.05, '
                                             'samples=14\n'
                                             '   iops        : min=301842, '
                                             'max=302954, avg=302405.50, '
                                             'stdev=344.06, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.02%, '
                                             '4=0.11%, 10=79.62%, 20=18.50%, '
                                             '50=1.70%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=17.59%, '
                                             'sys=80.17%, ctx=12447, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2116575,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=148MiB/s (155MB/s), '
                                             '148MiB/s-148MiB/s '
                                             '(155MB/s-155MB/s), io=1033MiB '
                                             '(1084MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.output'),
                                   'state': {'begin': 1666745068.7541783,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7893593311309814,
                                             'end': 1666745069.5435376,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_258.state')},
                       'cmd_259': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.output'),
                                   'state': {'begin': 1666745070.5367262,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.788844108581543,
                                             'end': 1666745071.3255703,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_259.state')},
                       'cmd_26': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.output'),
                                  'state': {'begin': 1666743952.9232578,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.471574783325195,
                                            'end': 1666743964.3948326,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_26.state')},
                       'cmd_260': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.output'),
                                   'state': {'begin': 1666745071.327926,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_io_uring-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.057929515838623,
                                             'end': 1666745082.3858554,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_260.state')},
                       'cmd_261': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752171,\n'
                                             '  "timestamp_ms" : '
                                             '1666752171569,\n'
                                             '  "time" : "Wed Oct 26 04:42:51 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1086826496,\n'
                                             '        "io_kbytes" : 1061354,\n'
                                             '        "bw_bytes" : 155238751,\n'
                                             '        "bw" : 151600,\n'
                                             '        "iops" : 303200.114269,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2122704,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1362,\n'
                                             '          "max" : 69741,\n'
                                             '          "mean" : 2801.643828,\n'
                                             '          "stddev" : '
                                             '1238.790601,\n'
                                             '          "N" : 2122705\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 124213,\n'
                                             '          "mean" : '
                                             '10150.495425,\n'
                                             '          "stddev" : '
                                             '2562.103546,\n'
                                             '          "N" : 2122707,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6880,\n'
                                             '            "5.000000" : 7968,\n'
                                             '            "10.000000" : 9408,\n'
                                             '            "20.000000" : 9536,\n'
                                             '            "30.000000" : 9664,\n'
                                             '            "40.000000" : 9664,\n'
                                             '            "50.000000" : 9792,\n'
                                             '            "60.000000" : 9792,\n'
                                             '            "70.000000" : 9920,\n'
                                             '            "80.000000" : 9920,\n'
                                             '            "90.000000" : '
                                             '11328,\n'
                                             '            "95.000000" : '
                                             '11840,\n'
                                             '            "99.000000" : '
                                             '23424,\n'
                                             '            "99.500000" : '
                                             '24960,\n'
                                             '            "99.900000" : '
                                             '30336,\n'
                                             '            "99.950000" : '
                                             '34560,\n'
                                             '            "99.990000" : 70144\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5861,\n'
                                             '          "max" : 125596,\n'
                                             '          "mean" : '
                                             '12952.138615,\n'
                                             '          "stddev" : '
                                             '2763.745120,\n'
                                             '          "N" : 2122707\n'
                                             '        },\n'
                                             '        "bw_min" : 151397,\n'
                                             '        "bw_max" : 151887,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '151658.000000,\n'
                                             '        "bw_dev" : 143.826864,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 302794,\n'
                                             '        "iops_max" : 303775,\n'
                                             '        "iops_mean" : '
                                             '303316.076923,\n'
                                             '        "iops_stddev" : '
                                             '287.786513,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.157143,\n'
                                             '      "sys_cpu" : 78.600000,\n'
                                             '      "ctx" : 12468,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.049371,\n'
                                             '        "4" : 0.088990,\n'
                                             '        "10" : 81.786580,\n'
                                             '        "20" : 16.352115,\n'
                                             '        "50" : 1.670982,\n'
                                             '        "100" : 0.044519,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142447: Wed '
                                             'Oct 26 04:42:51 2022\n'
                                             '  read: IOPS=303k, BW=148MiB/s '
                                             '(155MB/s)(1036MiB/7001msec)\n'
                                             '    slat (nsec): min=1362, '
                                             'max=69741, avg=2801.64, '
                                             'stdev=1238.79\n'
                                             '    clat (nsec): min=110, '
                                             'max=124213, avg=10150.50, '
                                             'stdev=2562.10\n'
                                             '     lat (usec): min=5, max=125, '
                                             'avg=12.95, stdev= 2.76\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6880],  '
                                             '5.00th=[ 7968], 10.00th=[ 9408], '
                                             '20.00th=[ 9536],\n'
                                             '     | 30.00th=[ 9664], '
                                             '40.00th=[ 9664], 50.00th=[ '
                                             '9792], 60.00th=[ 9792],\n'
                                             '     | 70.00th=[ 9920], '
                                             '80.00th=[ 9920], '
                                             '90.00th=[11328], '
                                             '95.00th=[11840],\n'
                                             '     | 99.00th=[23424], '
                                             '99.50th=[24960], '
                                             '99.90th=[30336], '
                                             '99.95th=[34560],\n'
                                             '     | 99.99th=[70144]\n'
                                             '   bw (  KiB/s): min=151397, '
                                             'max=151887, per=100.00%, '
                                             'avg=151658.00, stdev=143.83, '
                                             'samples=13\n'
                                             '   iops        : min=302794, '
                                             'max=303775, avg=303316.08, '
                                             'stdev=287.79, samples=13\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.05%, '
                                             '4=0.09%, 10=81.79%, 20=16.35%, '
                                             '50=1.67%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=19.16%, '
                                             'sys=78.60%, ctx=12468, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2122704,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=148MiB/s (155MB/s), '
                                             '148MiB/s-148MiB/s '
                                             '(155MB/s-155MB/s), io=1036MiB '
                                             '(1087MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.output'),
                                   'state': {'begin': 1666745082.3891363,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8009669780731201,
                                             'end': 1666745083.1901033,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_261.state')},
                       'cmd_262': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.output'),
                                   'state': {'begin': 1666745084.1825252,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7820878028869629,
                                             'end': 1666745084.964613,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_262.state')},
                       'cmd_263': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.output'),
                                   'state': {'begin': 1666745084.9666383,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.072291612625122,
                                             'end': 1666745096.03893,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_263.state')},
                       'cmd_264': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752185,\n'
                                             '  "timestamp_ms" : '
                                             '1666752185221,\n'
                                             '  "time" : "Wed Oct 26 04:43:05 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1041431040,\n'
                                             '        "io_kbytes" : 1017022,\n'
                                             '        "bw_bytes" : 148754612,\n'
                                             '        "bw" : 145268,\n'
                                             '        "iops" : 290535.780603,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2034041,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1403,\n'
                                             '          "max" : 42539,\n'
                                             '          "mean" : 2718.299399,\n'
                                             '          "stddev" : '
                                             '1303.395648,\n'
                                             '          "N" : 2034042\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4479,\n'
                                             '          "max" : 124985,\n'
                                             '          "mean" : '
                                             '10812.651804,\n'
                                             '          "stddev" : '
                                             '2728.741015,\n'
                                             '          "N" : 2034044,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 9024,\n'
                                             '            "5.000000" : 9024,\n'
                                             '            "10.000000" : 9024,\n'
                                             '            "20.000000" : 9152,\n'
                                             '            "30.000000" : 9280,\n'
                                             '            "40.000000" : '
                                             '10304,\n'
                                             '            "50.000000" : '
                                             '10432,\n'
                                             '            "60.000000" : '
                                             '10560,\n'
                                             '            "70.000000" : '
                                             '11584,\n'
                                             '            "80.000000" : '
                                             '11712,\n'
                                             '            "90.000000" : '
                                             '11840,\n'
                                             '            "95.000000" : '
                                             '12096,\n'
                                             '            "99.000000" : '
                                             '23680,\n'
                                             '            "99.500000" : '
                                             '28800,\n'
                                             '            "99.900000" : '
                                             '35072,\n'
                                             '            "99.950000" : '
                                             '40704,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 8055,\n'
                                             '          "max" : 129283,\n'
                                             '          "mean" : '
                                             '13530.950614,\n'
                                             '          "stddev" : '
                                             '2627.157009,\n'
                                             '          "N" : 2034044\n'
                                             '        },\n'
                                             '        "bw_min" : 144981,\n'
                                             '        "bw_max" : 145561,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '145329.000000,\n'
                                             '        "bw_dev" : 165.697593,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 289962,\n'
                                             '        "iops_max" : 291122,\n'
                                             '        "iops_mean" : '
                                             '290658.071429,\n'
                                             '        "iops_stddev" : '
                                             '331.438465,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 34.800000,\n'
                                             '      "sys_cpu" : 63.071429,\n'
                                             '      "ctx" : 12160,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 32.978195,\n'
                                             '        "20" : 65.542287,\n'
                                             '        "50" : 1.435320,\n'
                                             '        "100" : 0.043804,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2916402,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 15253,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 15253,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=142487: Wed '
                                             'Oct 26 04:43:05 2022\n'
                                             '  read: IOPS=291k, BW=142MiB/s '
                                             '(149MB/s)(993MiB/7001msec)\n'
                                             '    slat (nsec): min=1403, '
                                             'max=42539, avg=2718.30, '
                                             'stdev=1303.40\n'
                                             '    clat (usec): min=4, max=124, '
                                             'avg=10.81, stdev= 2.73\n'
                                             '     lat (usec): min=8, max=129, '
                                             'avg=13.53, stdev= 2.63\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 9024],  '
                                             '5.00th=[ 9024], 10.00th=[ 9024], '
                                             '20.00th=[ 9152],\n'
                                             '     | 30.00th=[ 9280], '
                                             '40.00th=[10304], '
                                             '50.00th=[10432], '
                                             '60.00th=[10560],\n'
                                             '     | 70.00th=[11584], '
                                             '80.00th=[11712], '
                                             '90.00th=[11840], '
                                             '95.00th=[12096],\n'
                                             '     | 99.00th=[23680], '
                                             '99.50th=[28800], '
                                             '99.90th=[35072], '
                                             '99.95th=[40704],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=144981, '
                                             'max=145561, per=100.00%, '
                                             'avg=145329.00, stdev=165.70, '
                                             'samples=14\n'
                                             '   iops        : min=289962, '
                                             'max=291122, avg=290658.07, '
                                             'stdev=331.44, samples=14\n'
                                             '  lat (usec)   : 10=32.98%, '
                                             '20=65.54%, 50=1.44%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=34.80%, '
                                             'sys=63.07%, ctx=12160, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2034041,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=142MiB/s (149MB/s), '
                                             '142MiB/s-142MiB/s '
                                             '(149MB/s-149MB/s), io=993MiB '
                                             '(1041MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2916402/0, '
                                             'merge=0/0, ticks=15253/0, '
                                             'in_queue=15253, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.output'),
                                   'state': {'begin': 1666745096.0416799,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7924807071685791,
                                             'end': 1666745096.8341606,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_264.state')},
                       'cmd_265': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.output'),
                                   'state': {'begin': 1666745097.8319213,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7851710319519043,
                                             'end': 1666745098.6170924,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_265.state')},
                       'cmd_266': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.output'),
                                   'state': {'begin': 1666745098.6191463,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.066049098968506,
                                             'end': 1666745109.6851954,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_266.state')},
                       'cmd_267': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752198,\n'
                                             '  "timestamp_ms" : '
                                             '1666752198867,\n'
                                             '  "time" : "Wed Oct 26 04:43:18 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1038159872,\n'
                                             '        "io_kbytes" : 1013828,\n'
                                             '        "bw_bytes" : 148287369,\n'
                                             '        "bw" : 144811,\n'
                                             '        "iops" : 289623.196686,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2027652,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1402,\n'
                                             '          "max" : 25027,\n'
                                             '          "mean" : 2728.420388,\n'
                                             '          "stddev" : '
                                             '1317.432382,\n'
                                             '          "N" : 2027653\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4689,\n'
                                             '          "max" : 93515,\n'
                                             '          "mean" : '
                                             '10845.511402,\n'
                                             '          "stddev" : '
                                             '2729.393787,\n'
                                             '          "N" : 2027655,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 9024,\n'
                                             '            "5.000000" : 9024,\n'
                                             '            "10.000000" : 9152,\n'
                                             '            "20.000000" : 9152,\n'
                                             '            "30.000000" : 9280,\n'
                                             '            "40.000000" : '
                                             '10304,\n'
                                             '            "50.000000" : '
                                             '10432,\n'
                                             '            "60.000000" : '
                                             '10816,\n'
                                             '            "70.000000" : '
                                             '11584,\n'
                                             '            "80.000000" : '
                                             '11712,\n'
                                             '            "90.000000" : '
                                             '11840,\n'
                                             '            "95.000000" : '
                                             '11968,\n'
                                             '            "99.000000" : '
                                             '23424,\n'
                                             '            "99.500000" : '
                                             '29056,\n'
                                             '            "99.900000" : '
                                             '35072,\n'
                                             '            "99.950000" : '
                                             '40192,\n'
                                             '            "99.990000" : 71168\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 8115,\n'
                                             '          "max" : 98064,\n'
                                             '          "mean" : '
                                             '13573.932355,\n'
                                             '          "stddev" : '
                                             '2591.494403,\n'
                                             '          "N" : 2027655\n'
                                             '        },\n'
                                             '        "bw_min" : 144516,\n'
                                             '        "bw_max" : 145113,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '144877.230769,\n'
                                             '        "bw_dev" : 161.265906,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 289032,\n'
                                             '        "iops_max" : 290226,\n'
                                             '        "iops_mean" : '
                                             '289754.692308,\n'
                                             '        "iops_stddev" : '
                                             '322.712820,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 34.757143,\n'
                                             '      "sys_cpu" : 63.128571,\n'
                                             '      "ctx" : 12125,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 36.527964,\n'
                                             '        "20" : 61.978979,\n'
                                             '        "50" : 1.450989,\n'
                                             '        "100" : 0.042216,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2908270,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 15201,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 15201,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=142526: Wed '
                                             'Oct 26 04:43:18 2022\n'
                                             '  read: IOPS=290k, BW=141MiB/s '
                                             '(148MB/s)(990MiB/7001msec)\n'
                                             '    slat (nsec): min=1402, '
                                             'max=25027, avg=2728.42, '
                                             'stdev=1317.43\n'
                                             '    clat (nsec): min=4689, '
                                             'max=93515, avg=10845.51, '
                                             'stdev=2729.39\n'
                                             '     lat (nsec): min=8115, '
                                             'max=98064, avg=13573.93, '
                                             'stdev=2591.49\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 9024],  '
                                             '5.00th=[ 9024], 10.00th=[ 9152], '
                                             '20.00th=[ 9152],\n'
                                             '     | 30.00th=[ 9280], '
                                             '40.00th=[10304], '
                                             '50.00th=[10432], '
                                             '60.00th=[10816],\n'
                                             '     | 70.00th=[11584], '
                                             '80.00th=[11712], '
                                             '90.00th=[11840], '
                                             '95.00th=[11968],\n'
                                             '     | 99.00th=[23424], '
                                             '99.50th=[29056], '
                                             '99.90th=[35072], '
                                             '99.95th=[40192],\n'
                                             '     | 99.99th=[71168]\n'
                                             '   bw (  KiB/s): min=144516, '
                                             'max=145113, per=100.00%, '
                                             'avg=144877.23, stdev=161.27, '
                                             'samples=13\n'
                                             '   iops        : min=289032, '
                                             'max=290226, avg=289754.69, '
                                             'stdev=322.71, samples=13\n'
                                             '  lat (usec)   : 10=36.53%, '
                                             '20=61.98%, 50=1.45%, 100=0.04%\n'
                                             '  cpu          : usr=34.76%, '
                                             'sys=63.13%, ctx=12125, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2027652,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=141MiB/s (148MB/s), '
                                             '141MiB/s-141MiB/s '
                                             '(148MB/s-148MB/s), io=990MiB '
                                             '(1038MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2908270/0, '
                                             'merge=0/0, ticks=15201/0, '
                                             'in_queue=15201, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.output'),
                                   'state': {'begin': 1666745109.687747,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7985000610351562,
                                             'end': 1666745110.486247,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_267.state')},
                       'cmd_268': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.output'),
                                   'state': {'begin': 1666745111.489625,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.783801794052124,
                                             'end': 1666745112.2734268,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_268.state')},
                       'cmd_269': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.output'),
                                   'state': {'begin': 1666745112.2754273,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.081075668334961,
                                             'end': 1666745123.356503,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_269.state')},
                       'cmd_27': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751053,\n'
                                            '  "timestamp_ms" : '
                                            '1666751053526,\n'
                                            '  "time" : "Wed Oct 26 04:24:13 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 439462400,\n'
                                            '        "io_kbytes" : 429162,\n'
                                            '        "bw_bytes" : 62771375,\n'
                                            '        "bw" : 61300,\n'
                                            '        "iops" : 122600.199971,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 858324,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 701,\n'
                                            '          "max" : 13997,\n'
                                            '          "mean" : 918.099893,\n'
                                            '          "stddev" : 66.379994,\n'
                                            '          "N" : 858325\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 551,\n'
                                            '          "max" : 101871,\n'
                                            '          "mean" : 6972.345603,\n'
                                            '          "stddev" : '
                                            '1116.060535,\n'
                                            '          "N" : 858324,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 6368,\n'
                                            '            "5.000000" : 6496,\n'
                                            '            "10.000000" : 6560,\n'
                                            '            "20.000000" : 6688,\n'
                                            '            "30.000000" : 6752,\n'
                                            '            "40.000000" : 6752,\n'
                                            '            "50.000000" : 6816,\n'
                                            '            "60.000000" : 6944,\n'
                                            '            "70.000000" : 6944,\n'
                                            '            "80.000000" : 7008,\n'
                                            '            "90.000000" : 7136,\n'
                                            '            "95.000000" : 7264,\n'
                                            '            "99.000000" : 12864,\n'
                                            '            "99.500000" : 13632,\n'
                                            '            "99.900000" : 16512,\n'
                                            '            "99.950000" : 16768,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7013,\n'
                                            '          "max" : 102792,\n'
                                            '          "mean" : 7890.445503,\n'
                                            '          "stddev" : '
                                            '1119.494292,\n'
                                            '          "N" : 858324\n'
                                            '        },\n'
                                            '        "bw_min" : 61255,\n'
                                            '        "bw_max" : 61417,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '61317.071429,\n'
                                            '        "bw_dev" : 43.492642,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 122510,\n'
                                            '        "iops_max" : 122834,\n'
                                            '        "iops_mean" : '
                                            '122634.214286,\n'
                                            '        "iops_stddev" : '
                                            '86.918334,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 99.985714,\n'
                                            '      "sys_cpu" : 0.000000,\n'
                                            '      "ctx" : 22,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.301574,\n'
                                            '        "20" : 1.679669,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014563,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-bdev_xnvme: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=139019: Wed Oct 26 04:24:13 '
                                            '2022\n'
                                            '  read: IOPS=123k, BW=59.9MiB/s '
                                            '(62.8MB/s)(419MiB/7001msec)\n'
                                            '    slat (nsec): min=701, '
                                            'max=13997, avg=918.10, '
                                            'stdev=66.38\n'
                                            '    clat (nsec): min=551, '
                                            'max=101871, avg=6972.35, '
                                            'stdev=1116.06\n'
                                            '     lat (usec): min=7, max=102, '
                                            'avg= 7.89, stdev= 1.12\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 6368],  5.00th=[ '
                                            '6496], 10.00th=[ 6560], 20.00th=[ '
                                            '6688],\n'
                                            '     | 30.00th=[ 6752], 40.00th=[ '
                                            '6752], 50.00th=[ 6816], 60.00th=[ '
                                            '6944],\n'
                                            '     | 70.00th=[ 6944], 80.00th=[ '
                                            '7008], 90.00th=[ 7136], 95.00th=[ '
                                            '7264],\n'
                                            '     | 99.00th=[12864], '
                                            '99.50th=[13632], 99.90th=[16512], '
                                            '99.95th=[16768],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=61255, '
                                            'max=61417, per=100.00%, '
                                            'avg=61317.07, stdev=43.49, '
                                            'samples=14\n'
                                            '   iops        : min=122510, '
                                            'max=122834, avg=122634.21, '
                                            'stdev=86.92, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%\n'
                                            '  lat (usec)   : 10=98.30%, '
                                            '20=1.68%, 50=0.01%, 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=99.99%, '
                                            'sys=0.00%, ctx=22, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=858324,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=59.9MiB/s (62.8MB/s), '
                                            '59.9MiB/s-59.9MiB/s '
                                            '(62.8MB/s-62.8MB/s), io=419MiB '
                                            '(439MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.output'),
                                  'state': {'begin': 1666743964.3977723,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7926888465881348,
                                            'end': 1666743965.1904612,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_27.state')},
                       'cmd_270': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752212,\n'
                                             '  "timestamp_ms" : '
                                             '1666752212538,\n'
                                             '  "time" : "Wed Oct 26 04:43:32 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1037223424,\n'
                                             '        "io_kbytes" : 1012913,\n'
                                             '        "bw_bytes" : 148153610,\n'
                                             '        "bw" : 144681,\n'
                                             '        "iops" : 289361.948293,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2025823,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1412,\n'
                                             '          "max" : 28864,\n'
                                             '          "mean" : 2730.817650,\n'
                                             '          "stddev" : '
                                             '1328.449286,\n'
                                             '          "N" : 2025824\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4679,\n'
                                             '          "max" : 134222,\n'
                                             '          "mean" : '
                                             '10856.180382,\n'
                                             '          "stddev" : '
                                             '2794.572731,\n'
                                             '          "N" : 2025826,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 9024,\n'
                                             '            "5.000000" : 9024,\n'
                                             '            "10.000000" : 9152,\n'
                                             '            "20.000000" : 9152,\n'
                                             '            "30.000000" : 9280,\n'
                                             '            "40.000000" : '
                                             '10304,\n'
                                             '            "50.000000" : '
                                             '10432,\n'
                                             '            "60.000000" : '
                                             '10816,\n'
                                             '            "70.000000" : '
                                             '11584,\n'
                                             '            "80.000000" : '
                                             '11712,\n'
                                             '            "90.000000" : '
                                             '11840,\n'
                                             '            "95.000000" : '
                                             '12096,\n'
                                             '            "99.000000" : '
                                             '23680,\n'
                                             '            "99.500000" : '
                                             '29056,\n'
                                             '            "99.900000" : '
                                             '35584,\n'
                                             '            "99.950000" : '
                                             '42240,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 8135,\n'
                                             '          "max" : 137117,\n'
                                             '          "mean" : '
                                             '13586.998051,\n'
                                             '          "stddev" : '
                                             '2658.191293,\n'
                                             '          "N" : 2025826\n'
                                             '        },\n'
                                             '        "bw_min" : 144341,\n'
                                             '        "bw_max" : 145442,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '144773.769231,\n'
                                             '        "bw_dev" : 340.786383,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 288682,\n'
                                             '        "iops_max" : 290884,\n'
                                             '        "iops_mean" : '
                                             '289547.461538,\n'
                                             '        "iops_stddev" : '
                                             '681.758341,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 35.828571,\n'
                                             '      "sys_cpu" : 62.028571,\n'
                                             '      "ctx" : 12148,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 36.357915,\n'
                                             '        "20" : 62.091604,\n'
                                             '        "50" : 1.503537,\n'
                                             '        "100" : 0.046253,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2832626,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 14860,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 14860,\n'
                                             '      "util" : 99.085923\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=142565: Wed '
                                             'Oct 26 04:43:32 2022\n'
                                             '  read: IOPS=289k, BW=141MiB/s '
                                             '(148MB/s)(989MiB/7001msec)\n'
                                             '    slat (nsec): min=1412, '
                                             'max=28864, avg=2730.82, '
                                             'stdev=1328.45\n'
                                             '    clat (usec): min=4, max=134, '
                                             'avg=10.86, stdev= 2.79\n'
                                             '     lat (usec): min=8, max=137, '
                                             'avg=13.59, stdev= 2.66\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 9024],  '
                                             '5.00th=[ 9024], 10.00th=[ 9152], '
                                             '20.00th=[ 9152],\n'
                                             '     | 30.00th=[ 9280], '
                                             '40.00th=[10304], '
                                             '50.00th=[10432], '
                                             '60.00th=[10816],\n'
                                             '     | 70.00th=[11584], '
                                             '80.00th=[11712], '
                                             '90.00th=[11840], '
                                             '95.00th=[12096],\n'
                                             '     | 99.00th=[23680], '
                                             '99.50th=[29056], '
                                             '99.90th=[35584], '
                                             '99.95th=[42240],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=144341, '
                                             'max=145442, per=100.00%, '
                                             'avg=144773.77, stdev=340.79, '
                                             'samples=13\n'
                                             '   iops        : min=288682, '
                                             'max=290884, avg=289547.46, '
                                             'stdev=681.76, samples=13\n'
                                             '  lat (usec)   : 10=36.36%, '
                                             '20=62.09%, 50=1.50%, 100=0.05%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=35.83%, '
                                             'sys=62.03%, ctx=12148, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2025823,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=141MiB/s (148MB/s), '
                                             '141MiB/s-141MiB/s '
                                             '(148MB/s-148MB/s), io=989MiB '
                                             '(1037MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2832626/0, '
                                             'merge=0/0, ticks=14860/0, '
                                             'in_queue=14860, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.output'),
                                   'state': {'begin': 1666745123.3594093,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7903752326965332,
                                             'end': 1666745124.1497846,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_270.state')},
                       'cmd_271': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.output'),
                                   'state': {'begin': 1666745126.1821842,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7924859523773193,
                                             'end': 1666745126.9746702,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_271.state')},
                       'cmd_272': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.output'),
                                   'state': {'begin': 1666745126.9771647,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_aio_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.554776906967163,
                                             'end': 1666745138.5319417,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_272.state')},
                       'cmd_273': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752227,\n'
                                             '  "timestamp_ms" : '
                                             '1666752227603,\n'
                                             '  "time" : "Wed Oct 26 04:43:47 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1071235584,\n'
                                             '        "io_kbytes" : 1046128,\n'
                                             '        "bw_bytes" : 153011796,\n'
                                             '        "bw" : 149425,\n'
                                             '        "iops" : 298850.878446,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2092255,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1583,\n'
                                             '          "max" : 29135,\n'
                                             '          "mean" : 2702.866753,\n'
                                             '          "stddev" : '
                                             '1007.331736,\n'
                                             '          "N" : 2092256\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 260,\n'
                                             '          "max" : 114535,\n'
                                             '          "mean" : 7822.386945,\n'
                                             '          "stddev" : '
                                             '3118.577266,\n'
                                             '          "N" : 2092257,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3280,\n'
                                             '            "5.000000" : 3376,\n'
                                             '            "10.000000" : 4512,\n'
                                             '            "20.000000" : 4832,\n'
                                             '            "30.000000" : 6432,\n'
                                             '            "40.000000" : 6688,\n'
                                             '            "50.000000" : 7200,\n'
                                             '            "60.000000" : 8512,\n'
                                             '            "70.000000" : 9792,\n'
                                             '            "80.000000" : '
                                             '10048,\n'
                                             '            "90.000000" : '
                                             '11072,\n'
                                             '            "95.000000" : '
                                             '11456,\n'
                                             '            "99.000000" : '
                                             '18560,\n'
                                             '            "99.500000" : '
                                             '21376,\n'
                                             '            "99.900000" : '
                                             '24960,\n'
                                             '            "99.950000" : '
                                             '31872,\n'
                                             '            "99.990000" : 63744\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5791,\n'
                                             '          "max" : 116228,\n'
                                             '          "mean" : '
                                             '10525.253804,\n'
                                             '          "stddev" : '
                                             '3235.371221,\n'
                                             '          "N" : 2092257\n'
                                             '        },\n'
                                             '        "bw_min" : 149025,\n'
                                             '        "bw_max" : 149805,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '149469.785714,\n'
                                             '        "bw_dev" : 290.944134,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 298050,\n'
                                             '        "iops_max" : 299606,\n'
                                             '        "iops_mean" : '
                                             '298939.571429,\n'
                                             '        "iops_stddev" : '
                                             '581.128440,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 43.357143,\n'
                                             '      "sys_cpu" : 56.614286,\n'
                                             '      "ctx" : 22,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.967043,\n'
                                             '        "2" : 60.186784,\n'
                                             '        "4" : 38.846173,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.128904,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.015390,\n'
                                             '        "4" : 7.186170,\n'
                                             '        "10" : 73.121154,\n'
                                             '        "20" : 18.881733,\n'
                                             '        "50" : 0.629417,\n'
                                             '        "100" : 0.036946,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=142613: Wed '
                                             'Oct 26 04:43:47 2022\n'
                                             '  read: IOPS=299k, BW=146MiB/s '
                                             '(153MB/s)(1022MiB/7001msec)\n'
                                             '    slat (nsec): min=1583, '
                                             'max=29135, avg=2702.87, '
                                             'stdev=1007.33\n'
                                             '    clat (nsec): min=260, '
                                             'max=114535, avg=7822.39, '
                                             'stdev=3118.58\n'
                                             '     lat (usec): min=5, max=116, '
                                             'avg=10.53, stdev= 3.24\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3280],  '
                                             '5.00th=[ 3376], 10.00th=[ 4512], '
                                             '20.00th=[ 4832],\n'
                                             '     | 30.00th=[ 6432], '
                                             '40.00th=[ 6688], 50.00th=[ '
                                             '7200], 60.00th=[ 8512],\n'
                                             '     | 70.00th=[ 9792], '
                                             '80.00th=[10048], '
                                             '90.00th=[11072], '
                                             '95.00th=[11456],\n'
                                             '     | 99.00th=[18560], '
                                             '99.50th=[21376], '
                                             '99.90th=[24960], '
                                             '99.95th=[31872],\n'
                                             '     | 99.99th=[63744]\n'
                                             '   bw (  KiB/s): min=149025, '
                                             'max=149805, per=100.00%, '
                                             'avg=149469.79, stdev=290.94, '
                                             'samples=14\n'
                                             '   iops        : min=298050, '
                                             'max=299606, avg=298939.57, '
                                             'stdev=581.13, samples=14\n'
                                             '  lat (nsec)   : 500=0.13%, '
                                             '750=0.01%, 1000=0.01%\n'
                                             '  lat (usec)   : 2=0.02%, '
                                             '4=7.19%, 10=73.12%, 20=18.88%, '
                                             '50=0.63%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=43.36%, '
                                             'sys=56.61%, ctx=22, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=1.0%, '
                                             '2=60.2%, 4=38.8%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2092255,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=146MiB/s (153MB/s), '
                                             '146MiB/s-146MiB/s '
                                             '(153MB/s-153MB/s), io=1022MiB '
                                             '(1071MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.output'),
                                   'state': {'begin': 1666745138.535005,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7863025665283203,
                                             'end': 1666745139.3213077,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_273.state')},
                       'cmd_274': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.output'),
                                   'state': {'begin': 1666745141.3469138,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7810285091400146,
                                             'end': 1666745142.1279423,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_274.state')},
                       'cmd_275': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.output'),
                                   'state': {'begin': 1666745142.130097,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_aio_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.5094473361969,
                                             'end': 1666745153.6395442,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_275.state')},
                       'cmd_276': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752242,\n'
                                             '  "timestamp_ms" : '
                                             '1666752242721,\n'
                                             '  "time" : "Wed Oct 26 04:44:02 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1061995520,\n'
                                             '        "io_kbytes" : 1037105,\n'
                                             '        "bw_bytes" : 151691975,\n'
                                             '        "bw" : 148136,\n'
                                             '        "iops" : 296273.103842,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2074208,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1603,\n'
                                             '          "max" : 57037,\n'
                                             '          "mean" : 2788.654794,\n'
                                             '          "stddev" : '
                                             '1082.503450,\n'
                                             '          "N" : 2074209\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 241,\n'
                                             '          "max" : 157215,\n'
                                             '          "mean" : 7732.501443,\n'
                                             '          "stddev" : '
                                             '3123.606252,\n'
                                             '          "N" : 2074210,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3312,\n'
                                             '            "5.000000" : 4512,\n'
                                             '            "10.000000" : 4576,\n'
                                             '            "20.000000" : 4704,\n'
                                             '            "30.000000" : 6432,\n'
                                             '            "40.000000" : 6560,\n'
                                             '            "50.000000" : 6880,\n'
                                             '            "60.000000" : 8256,\n'
                                             '            "70.000000" : 9920,\n'
                                             '            "80.000000" : '
                                             '10048,\n'
                                             '            "90.000000" : '
                                             '11200,\n'
                                             '            "95.000000" : '
                                             '11456,\n'
                                             '            "99.000000" : '
                                             '18304,\n'
                                             '            "99.500000" : '
                                             '21120,\n'
                                             '            "99.900000" : '
                                             '25216,\n'
                                             '            "99.950000" : '
                                             '31872,\n'
                                             '            "99.990000" : 62720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5721,\n'
                                             '          "max" : 158908,\n'
                                             '          "mean" : '
                                             '10521.156284,\n'
                                             '          "stddev" : '
                                             '3475.499276,\n'
                                             '          "N" : 2074210\n'
                                             '        },\n'
                                             '        "bw_min" : 147810,\n'
                                             '        "bw_max" : 148550,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '148173.714286,\n'
                                             '        "bw_dev" : 244.282443,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 295620,\n'
                                             '        "iops_max" : 297100,\n'
                                             '        "iops_mean" : '
                                             '296347.428571,\n'
                                             '        "iops_stddev" : '
                                             '488.564887,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 43.757143,\n'
                                             '      "sys_cpu" : 56.214286,\n'
                                             '      "ctx" : 23,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.941564,\n'
                                             '        "2" : 61.963410,\n'
                                             '        "4" : 37.095026,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.128868,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.017645,\n'
                                             '        "4" : 4.562320,\n'
                                             '        "10" : 70.359482,\n'
                                             '        "20" : 24.313714,\n'
                                             '        "50" : 0.580125,\n'
                                             '        "100" : 0.036978,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=142662: Wed '
                                             'Oct 26 04:44:02 2022\n'
                                             '  read: IOPS=296k, BW=145MiB/s '
                                             '(152MB/s)(1013MiB/7001msec)\n'
                                             '    slat (nsec): min=1603, '
                                             'max=57037, avg=2788.65, '
                                             'stdev=1082.50\n'
                                             '    clat (nsec): min=241, '
                                             'max=157215, avg=7732.50, '
                                             'stdev=3123.61\n'
                                             '     lat (usec): min=5, max=158, '
                                             'avg=10.52, stdev= 3.48\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3312],  '
                                             '5.00th=[ 4512], 10.00th=[ 4576], '
                                             '20.00th=[ 4704],\n'
                                             '     | 30.00th=[ 6432], '
                                             '40.00th=[ 6560], 50.00th=[ '
                                             '6880], 60.00th=[ 8256],\n'
                                             '     | 70.00th=[ 9920], '
                                             '80.00th=[10048], '
                                             '90.00th=[11200], '
                                             '95.00th=[11456],\n'
                                             '     | 99.00th=[18304], '
                                             '99.50th=[21120], '
                                             '99.90th=[25216], '
                                             '99.95th=[31872],\n'
                                             '     | 99.99th=[62720]\n'
                                             '   bw (  KiB/s): min=147810, '
                                             'max=148550, per=100.00%, '
                                             'avg=148173.71, stdev=244.28, '
                                             'samples=14\n'
                                             '   iops        : min=295620, '
                                             'max=297100, avg=296347.43, '
                                             'stdev=488.56, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.13%, 750=0.01%\n'
                                             '  lat (usec)   : 2=0.02%, '
                                             '4=4.56%, 10=70.36%, 20=24.31%, '
                                             '50=0.58%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=43.76%, '
                                             'sys=56.21%, ctx=23, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.9%, '
                                             '2=62.0%, 4=37.1%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2074208,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=145MiB/s (152MB/s), '
                                             '145MiB/s-145MiB/s '
                                             '(152MB/s-152MB/s), io=1013MiB '
                                             '(1062MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.output'),
                                   'state': {'begin': 1666745153.6424088,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7868928909301758,
                                             'end': 1666745154.4293017,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_276.state')},
                       'cmd_277': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.output'),
                                   'state': {'begin': 1666745156.4354622,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7887165546417236,
                                             'end': 1666745157.2241788,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_277.state')},
                       'cmd_278': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.output'),
                                   'state': {'begin': 1666745157.2262728,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_aio_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.493136167526245,
                                             'end': 1666745168.719409,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_278.state')},
                       'cmd_279': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752257,\n'
                                             '  "timestamp_ms" : '
                                             '1666752257808,\n'
                                             '  "time" : "Wed Oct 26 04:44:17 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1064985600,\n'
                                             '        "io_kbytes" : 1040025,\n'
                                             '        "bw_bytes" : 152119068,\n'
                                             '        "bw" : 148553,\n'
                                             '        "iops" : 297107.270390,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2080048,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1593,\n'
                                             '          "max" : 32571,\n'
                                             '          "mean" : 2739.548837,\n'
                                             '          "stddev" : '
                                             '1047.822313,\n'
                                             '          "N" : 2080049\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 270,\n'
                                             '          "max" : 117951,\n'
                                             '          "mean" : 7801.096301,\n'
                                             '          "stddev" : '
                                             '3165.549384,\n'
                                             '          "N" : 2080050,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3312,\n'
                                             '            "5.000000" : 3440,\n'
                                             '            "10.000000" : 4576,\n'
                                             '            "20.000000" : 4704,\n'
                                             '            "30.000000" : 6432,\n'
                                             '            "40.000000" : 6560,\n'
                                             '            "50.000000" : 6944,\n'
                                             '            "60.000000" : 8512,\n'
                                             '            "70.000000" : 9920,\n'
                                             '            "80.000000" : '
                                             '10048,\n'
                                             '            "90.000000" : '
                                             '11200,\n'
                                             '            "95.000000" : '
                                             '11456,\n'
                                             '            "99.000000" : '
                                             '18304,\n'
                                             '            "99.500000" : '
                                             '21120,\n'
                                             '            "99.900000" : '
                                             '25472,\n'
                                             '            "99.950000" : '
                                             '32128,\n'
                                             '            "99.990000" : 64256\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5830,\n'
                                             '          "max" : 119664,\n'
                                             '          "mean" : '
                                             '10540.645231,\n'
                                             '          "stddev" : '
                                             '3389.356523,\n'
                                             '          "N" : 2080050\n'
                                             '        },\n'
                                             '        "bw_min" : 148084,\n'
                                             '        "bw_max" : 149034,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '148592.357143,\n'
                                             '        "bw_dev" : 285.577043,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 296168,\n'
                                             '        "iops_max" : 298068,\n'
                                             '        "iops_mean" : '
                                             '297184.714286,\n'
                                             '        "iops_stddev" : '
                                             '571.154085,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 41.985714,\n'
                                             '      "sys_cpu" : 57.985714,\n'
                                             '      "ctx" : 25,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.959641,\n'
                                             '        "2" : 61.257433,\n'
                                             '        "4" : 37.782926,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.123459,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.015096,\n'
                                             '        "4" : 6.310912,\n'
                                             '        "10" : 69.856369,\n'
                                             '        "20" : 23.058602,\n'
                                             '        "50" : 0.596188,\n'
                                             '        "100" : 0.038797,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=142712: Wed '
                                             'Oct 26 04:44:17 2022\n'
                                             '  read: IOPS=297k, BW=145MiB/s '
                                             '(152MB/s)(1016MiB/7001msec)\n'
                                             '    slat (nsec): min=1593, '
                                             'max=32571, avg=2739.55, '
                                             'stdev=1047.82\n'
                                             '    clat (nsec): min=270, '
                                             'max=117951, avg=7801.10, '
                                             'stdev=3165.55\n'
                                             '     lat (usec): min=5, max=119, '
                                             'avg=10.54, stdev= 3.39\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3312],  '
                                             '5.00th=[ 3440], 10.00th=[ 4576], '
                                             '20.00th=[ 4704],\n'
                                             '     | 30.00th=[ 6432], '
                                             '40.00th=[ 6560], 50.00th=[ '
                                             '6944], 60.00th=[ 8512],\n'
                                             '     | 70.00th=[ 9920], '
                                             '80.00th=[10048], '
                                             '90.00th=[11200], '
                                             '95.00th=[11456],\n'
                                             '     | 99.00th=[18304], '
                                             '99.50th=[21120], '
                                             '99.90th=[25472], '
                                             '99.95th=[32128],\n'
                                             '     | 99.99th=[64256]\n'
                                             '   bw (  KiB/s): min=148084, '
                                             'max=149034, per=100.00%, '
                                             'avg=148592.36, stdev=285.58, '
                                             'samples=14\n'
                                             '   iops        : min=296168, '
                                             'max=298068, avg=297184.71, '
                                             'stdev=571.15, samples=14\n'
                                             '  lat (nsec)   : 500=0.12%\n'
                                             '  lat (usec)   : 2=0.02%, '
                                             '4=6.31%, 10=69.86%, 20=23.06%, '
                                             '50=0.60%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=41.99%, '
                                             'sys=57.99%, ctx=25, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=1.0%, '
                                             '2=61.3%, 4=37.8%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2080048,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=145MiB/s (152MB/s), '
                                             '145MiB/s-145MiB/s '
                                             '(152MB/s-152MB/s), io=1016MiB '
                                             '(1065MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.output'),
                                   'state': {'begin': 1666745168.7224128,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7964358329772949,
                                             'end': 1666745169.5188487,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_279.state')},
                       'cmd_28': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.output'),
                                  'state': {'begin': 1666743966.1824296,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7796878814697266,
                                            'end': 1666743966.9621174,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_28.state')},
                       'cmd_280': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.output'),
                                   'state': {'begin': 1666745171.658184,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7816324234008789,
                                             'end': 1666745172.4398165,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_280.state')},
                       'cmd_281': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.output'),
                                   'state': {'begin': 1666745172.4418705,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.521850824356079,
                                             'end': 1666745183.9637213,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_281.state')},
                       'cmd_282': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752273,\n'
                                             '  "timestamp_ms" : '
                                             '1666752273038,\n'
                                             '  "time" : "Wed Oct 26 04:44:33 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1012254720,\n'
                                             '        "io_kbytes" : 988530,\n'
                                             '        "bw_bytes" : 144587161,\n'
                                             '        "bw" : 141198,\n'
                                             '        "iops" : 282396.514784,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1977058,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1603,\n'
                                             '          "max" : 63850,\n'
                                             '          "mean" : 2849.242207,\n'
                                             '          "stddev" : '
                                             '1275.330036,\n'
                                             '          "N" : 1977059\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 641,\n'
                                             '          "max" : 100899,\n'
                                             '          "mean" : 8513.515201,\n'
                                             '          "stddev" : '
                                             '2759.933150,\n'
                                             '          "N" : 1977060,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4832,\n'
                                             '            "5.000000" : 4960,\n'
                                             '            "10.000000" : 5088,\n'
                                             '            "20.000000" : 6880,\n'
                                             '            "30.000000" : 7456,\n'
                                             '            "40.000000" : 7584,\n'
                                             '            "50.000000" : 7648,\n'
                                             '            "60.000000" : 9280,\n'
                                             '            "70.000000" : 9408,\n'
                                             '            "80.000000" : 9536,\n'
                                             '            "90.000000" : '
                                             '11328,\n'
                                             '            "95.000000" : '
                                             '12224,\n'
                                             '            "99.000000" : '
                                             '18560,\n'
                                             '            "99.500000" : '
                                             '21120,\n'
                                             '            "99.900000" : '
                                             '25216,\n'
                                             '            "99.950000" : '
                                             '31872,\n'
                                             '            "99.990000" : 63744\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6052,\n'
                                             '          "max" : 102622,\n'
                                             '          "mean" : '
                                             '11362.757426,\n'
                                             '          "stddev" : '
                                             '3379.702263,\n'
                                             '          "N" : 1977060\n'
                                             '        },\n'
                                             '        "bw_min" : 140715,\n'
                                             '        "bw_max" : 141758,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '141236.428571,\n'
                                             '        "bw_dev" : 312.103216,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 281430,\n'
                                             '        "iops_max" : 283516,\n'
                                             '        "iops_mean" : '
                                             '282472.928571,\n'
                                             '        "iops_stddev" : '
                                             '624.256415,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 43.642857,\n'
                                             '      "sys_cpu" : 56.328571,\n'
                                             '      "ctx" : 24,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.877010,\n'
                                             '        "2" : 56.256974,\n'
                                             '        "4" : 42.866016,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.053918,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.015680,\n'
                                             '        "4" : 0.791783,\n'
                                             '        "10" : 82.251962,\n'
                                             '        "20" : 16.201093,\n'
                                             '        "50" : 0.643279,\n'
                                             '        "100" : 0.037986,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142761: Wed '
                                             'Oct 26 04:44:33 2022\n'
                                             '  read: IOPS=282k, BW=138MiB/s '
                                             '(145MB/s)(965MiB/7001msec)\n'
                                             '    slat (nsec): min=1603, '
                                             'max=63850, avg=2849.24, '
                                             'stdev=1275.33\n'
                                             '    clat (nsec): min=641, '
                                             'max=100899, avg=8513.52, '
                                             'stdev=2759.93\n'
                                             '     lat (usec): min=6, max=102, '
                                             'avg=11.36, stdev= 3.38\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4832],  '
                                             '5.00th=[ 4960], 10.00th=[ 5088], '
                                             '20.00th=[ 6880],\n'
                                             '     | 30.00th=[ 7456], '
                                             '40.00th=[ 7584], 50.00th=[ '
                                             '7648], 60.00th=[ 9280],\n'
                                             '     | 70.00th=[ 9408], '
                                             '80.00th=[ 9536], '
                                             '90.00th=[11328], '
                                             '95.00th=[12224],\n'
                                             '     | 99.00th=[18560], '
                                             '99.50th=[21120], '
                                             '99.90th=[25216], '
                                             '99.95th=[31872],\n'
                                             '     | 99.99th=[63744]\n'
                                             '   bw (  KiB/s): min=140715, '
                                             'max=141758, per=100.00%, '
                                             'avg=141236.43, stdev=312.10, '
                                             'samples=14\n'
                                             '   iops        : min=281430, '
                                             'max=283516, avg=282472.93, '
                                             'stdev=624.26, samples=14\n'
                                             '  lat (nsec)   : 750=0.05%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 2=0.02%, '
                                             '4=0.79%, 10=82.25%, 20=16.20%, '
                                             '50=0.64%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=43.64%, '
                                             'sys=56.33%, ctx=24, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.9%, '
                                             '2=56.3%, 4=42.9%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1977058,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=138MiB/s (145MB/s), '
                                             '138MiB/s-138MiB/s '
                                             '(145MB/s-145MB/s), io=965MiB '
                                             '(1012MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.output'),
                                   'state': {'begin': 1666745183.9666882,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7833597660064697,
                                             'end': 1666745184.750048,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_282.state')},
                       'cmd_283': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.output'),
                                   'state': {'begin': 1666745186.770997,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7856574058532715,
                                             'end': 1666745187.5566545,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_283.state')},
                       'cmd_284': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.output'),
                                   'state': {'begin': 1666745187.558708,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.544718265533447,
                                             'end': 1666745199.1034262,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_284.state')},
                       'cmd_285': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752288,\n'
                                             '  "timestamp_ms" : '
                                             '1666752288178,\n'
                                             '  "time" : "Wed Oct 26 04:44:48 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1009187328,\n'
                                             '        "io_kbytes" : 985534,\n'
                                             '        "bw_bytes" : 144149025,\n'
                                             '        "bw" : 140770,\n'
                                             '        "iops" : 281540.494215,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1971065,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1613,\n'
                                             '          "max" : 36499,\n'
                                             '          "mean" : 2864.161118,\n'
                                             '          "stddev" : '
                                             '1277.898171,\n'
                                             '          "N" : 1971066\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 641,\n'
                                             '          "max" : 123301,\n'
                                             '          "mean" : 8534.661721,\n'
                                             '          "stddev" : '
                                             '2698.421230,\n'
                                             '          "N" : 1971065,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4896,\n'
                                             '            "5.000000" : 5024,\n'
                                             '            "10.000000" : 5152,\n'
                                             '            "20.000000" : 6944,\n'
                                             '            "30.000000" : 7456,\n'
                                             '            "40.000000" : 7584,\n'
                                             '            "50.000000" : 7712,\n'
                                             '            "60.000000" : 9280,\n'
                                             '            "70.000000" : 9408,\n'
                                             '            "80.000000" : 9536,\n'
                                             '            "90.000000" : '
                                             '10944,\n'
                                             '            "95.000000" : '
                                             '12224,\n'
                                             '            "99.000000" : '
                                             '18304,\n'
                                             '            "99.500000" : '
                                             '21120,\n'
                                             '            "99.900000" : '
                                             '25472,\n'
                                             '            "99.950000" : '
                                             '32384,\n'
                                             '            "99.990000" : 63744\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6042,\n'
                                             '          "max" : 127870,\n'
                                             '          "mean" : '
                                             '11398.823449,\n'
                                             '          "stddev" : '
                                             '3337.911286,\n'
                                             '          "N" : 1971065\n'
                                             '        },\n'
                                             '        "bw_min" : 139935,\n'
                                             '        "bw_max" : 141743,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '140813.428571,\n'
                                             '        "bw_dev" : 451.996207,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 279870,\n'
                                             '        "iops_max" : 283486,\n'
                                             '        "iops_mean" : '
                                             '281626.928571,\n'
                                             '        "iops_stddev" : '
                                             '904.069642,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 44.785714,\n'
                                             '      "sys_cpu" : 55.185714,\n'
                                             '      "ctx" : 22,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.869327,\n'
                                             '        "2" : 55.711506,\n'
                                             '        "4" : 43.419167,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.051191,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.016691,\n'
                                             '        "4" : 0.692570,\n'
                                             '        "10" : 83.599323,\n'
                                             '        "20" : 14.998186,\n'
                                             '        "50" : 0.595871,\n'
                                             '        "100" : 0.041500,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142811: Wed '
                                             'Oct 26 04:44:48 2022\n'
                                             '  read: IOPS=282k, BW=137MiB/s '
                                             '(144MB/s)(962MiB/7001msec)\n'
                                             '    slat (nsec): min=1613, '
                                             'max=36499, avg=2864.16, '
                                             'stdev=1277.90\n'
                                             '    clat (nsec): min=641, '
                                             'max=123301, avg=8534.66, '
                                             'stdev=2698.42\n'
                                             '     lat (usec): min=6, max=127, '
                                             'avg=11.40, stdev= 3.34\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4896],  '
                                             '5.00th=[ 5024], 10.00th=[ 5152], '
                                             '20.00th=[ 6944],\n'
                                             '     | 30.00th=[ 7456], '
                                             '40.00th=[ 7584], 50.00th=[ '
                                             '7712], 60.00th=[ 9280],\n'
                                             '     | 70.00th=[ 9408], '
                                             '80.00th=[ 9536], '
                                             '90.00th=[10944], '
                                             '95.00th=[12224],\n'
                                             '     | 99.00th=[18304], '
                                             '99.50th=[21120], '
                                             '99.90th=[25472], '
                                             '99.95th=[32384],\n'
                                             '     | 99.99th=[63744]\n'
                                             '   bw (  KiB/s): min=139935, '
                                             'max=141743, per=100.00%, '
                                             'avg=140813.43, stdev=452.00, '
                                             'samples=14\n'
                                             '   iops        : min=279870, '
                                             'max=283486, avg=281626.93, '
                                             'stdev=904.07, samples=14\n'
                                             '  lat (nsec)   : 750=0.05%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 2=0.02%, '
                                             '4=0.69%, 10=83.60%, 20=15.00%, '
                                             '50=0.60%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=44.79%, '
                                             'sys=55.19%, ctx=22, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.9%, '
                                             '2=55.7%, 4=43.4%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1971065,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=137MiB/s (144MB/s), '
                                             '137MiB/s-137MiB/s '
                                             '(144MB/s-144MB/s), io=962MiB '
                                             '(1009MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.output'),
                                   'state': {'begin': 1666745199.106262,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7916183471679688,
                                             'end': 1666745199.8978803,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_285.state')},
                       'cmd_286': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.output'),
                                   'state': {'begin': 1666745201.9087136,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7794864177703857,
                                             'end': 1666745202.6882,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_286.state')},
                       'cmd_287': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.output'),
                                   'state': {'begin': 1666745202.6904583,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.516079425811768,
                                             'end': 1666745214.2065377,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_287.state')},
                       'cmd_288': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752303,\n'
                                             '  "timestamp_ms" : '
                                             '1666752303292,\n'
                                             '  "time" : "Wed Oct 26 04:45:03 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1011238912,\n'
                                             '        "io_kbytes" : 987538,\n'
                                             '        "bw_bytes" : 144442067,\n'
                                             '        "bw" : 141056,\n'
                                             '        "iops" : 282112.841023,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1975072,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1613,\n'
                                             '          "max" : 61526,\n'
                                             '          "mean" : 2852.995997,\n'
                                             '          "stddev" : '
                                             '1262.832611,\n'
                                             '          "N" : 1975073\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 631,\n'
                                             '          "max" : 118362,\n'
                                             '          "mean" : 8518.367863,\n'
                                             '          "stddev" : '
                                             '2664.843579,\n'
                                             '          "N" : 1975072,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 4832,\n'
                                             '            "5.000000" : 4960,\n'
                                             '            "10.000000" : 5088,\n'
                                             '            "20.000000" : 6880,\n'
                                             '            "30.000000" : 7456,\n'
                                             '            "40.000000" : 7584,\n'
                                             '            "50.000000" : 7712,\n'
                                             '            "60.000000" : 9280,\n'
                                             '            "70.000000" : 9408,\n'
                                             '            "80.000000" : 9536,\n'
                                             '            "90.000000" : '
                                             '10944,\n'
                                             '            "95.000000" : '
                                             '12224,\n'
                                             '            "99.000000" : '
                                             '18304,\n'
                                             '            "99.500000" : '
                                             '20864,\n'
                                             '            "99.900000" : '
                                             '24960,\n'
                                             '            "99.950000" : '
                                             '30592,\n'
                                             '            "99.990000" : 62208\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6021,\n'
                                             '          "max" : 120075,\n'
                                             '          "mean" : '
                                             '11371.363823,\n'
                                             '          "stddev" : '
                                             '3297.605000,\n'
                                             '          "N" : 1975072\n'
                                             '        },\n'
                                             '        "bw_min" : 140440,\n'
                                             '        "bw_max" : 142061,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '141095.714286,\n'
                                             '        "bw_dev" : 440.312377,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 280880,\n'
                                             '        "iops_max" : 284122,\n'
                                             '        "iops_mean" : '
                                             '282191.428571,\n'
                                             '        "iops_stddev" : '
                                             '880.624753,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 43.471429,\n'
                                             '      "sys_cpu" : 56.500000,\n'
                                             '      "ctx" : 20,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.861437,\n'
                                             '        "2" : 55.847635,\n'
                                             '        "4" : 43.290928,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.052150,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.018075,\n'
                                             '        "4" : 0.862348,\n'
                                             '        "10" : 82.898092,\n'
                                             '        "20" : 15.549256,\n'
                                             '        "50" : 0.580789,\n'
                                             '        "100" : 0.035391,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142860: Wed '
                                             'Oct 26 04:45:03 2022\n'
                                             '  read: IOPS=282k, BW=138MiB/s '
                                             '(144MB/s)(964MiB/7001msec)\n'
                                             '    slat (nsec): min=1613, '
                                             'max=61526, avg=2853.00, '
                                             'stdev=1262.83\n'
                                             '    clat (nsec): min=631, '
                                             'max=118362, avg=8518.37, '
                                             'stdev=2664.84\n'
                                             '     lat (usec): min=6, max=120, '
                                             'avg=11.37, stdev= 3.30\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 4832],  '
                                             '5.00th=[ 4960], 10.00th=[ 5088], '
                                             '20.00th=[ 6880],\n'
                                             '     | 30.00th=[ 7456], '
                                             '40.00th=[ 7584], 50.00th=[ '
                                             '7712], 60.00th=[ 9280],\n'
                                             '     | 70.00th=[ 9408], '
                                             '80.00th=[ 9536], '
                                             '90.00th=[10944], '
                                             '95.00th=[12224],\n'
                                             '     | 99.00th=[18304], '
                                             '99.50th=[20864], '
                                             '99.90th=[24960], '
                                             '99.95th=[30592],\n'
                                             '     | 99.99th=[62208]\n'
                                             '   bw (  KiB/s): min=140440, '
                                             'max=142061, per=100.00%, '
                                             'avg=141095.71, stdev=440.31, '
                                             'samples=14\n'
                                             '   iops        : min=280880, '
                                             'max=284122, avg=282191.43, '
                                             'stdev=880.62, samples=14\n'
                                             '  lat (nsec)   : 750=0.05%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 2=0.02%, '
                                             '4=0.86%, 10=82.90%, 20=15.55%, '
                                             '50=0.58%\n'
                                             '  lat (usec)   : 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=43.47%, '
                                             'sys=56.50%, ctx=20, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.9%, '
                                             '2=55.8%, 4=43.3%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1975072,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=138MiB/s (144MB/s), '
                                             '138MiB/s-138MiB/s '
                                             '(144MB/s-144MB/s), io=964MiB '
                                             '(1011MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.output'),
                                   'state': {'begin': 1666745214.209567,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7944097518920898,
                                             'end': 1666745215.0039768,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_288.state')},
                       'cmd_289': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.output'),
                                   'state': {'begin': 1666745215.9990199,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7896780967712402,
                                             'end': 1666745216.788698,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_289.state')},
                       'cmd_29': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.output'),
                                  'state': {'begin': 1666743966.9641044,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-reference_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.041018009185791,
                                            'end': 1666743978.0051224,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_29.state')},
                       'cmd_290': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.output'),
                                   'state': {'begin': 1666745216.7908251,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.089628458023071,
                                             'end': 1666745227.8804536,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_290.state')},
                       'cmd_291': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752317,\n'
                                             '  "timestamp_ms" : '
                                             '1666752317061,\n'
                                             '  "time" : "Wed Oct 26 04:45:17 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 995397120,\n'
                                             '        "io_kbytes" : 972067,\n'
                                             '        "bw_bytes" : 142179277,\n'
                                             '        "bw" : 138846,\n'
                                             '        "iops" : 277693.329524,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1944131,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1472,\n'
                                             '          "max" : 23805,\n'
                                             '          "mean" : 2275.900375,\n'
                                             '          "stddev" : '
                                             '900.413544,\n'
                                             '          "N" : 1944132\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4949,\n'
                                             '          "max" : 133701,\n'
                                             '          "mean" : '
                                             '11331.964851,\n'
                                             '          "stddev" : '
                                             '2845.799578,\n'
                                             '          "N" : 1944134,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 9408,\n'
                                             '            "5.000000" : 9536,\n'
                                             '            "10.000000" : 9536,\n'
                                             '            "20.000000" : 9664,\n'
                                             '            "30.000000" : 9664,\n'
                                             '            "40.000000" : 9792,\n'
                                             '            "50.000000" : '
                                             '11072,\n'
                                             '            "60.000000" : '
                                             '12096,\n'
                                             '            "70.000000" : '
                                             '12224,\n'
                                             '            "80.000000" : '
                                             '12352,\n'
                                             '            "90.000000" : '
                                             '12352,\n'
                                             '            "95.000000" : '
                                             '12608,\n'
                                             '            "99.000000" : '
                                             '23680,\n'
                                             '            "99.500000" : '
                                             '28288,\n'
                                             '            "99.900000" : '
                                             '36096,\n'
                                             '            "99.950000" : '
                                             '40192,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 7484,\n'
                                             '          "max" : 136586,\n'
                                             '          "mean" : '
                                             '13607.865899,\n'
                                             '          "stddev" : '
                                             '2632.972492,\n'
                                             '          "N" : 1944134\n'
                                             '        },\n'
                                             '        "bw_min" : 138554,\n'
                                             '        "bw_max" : 139440,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '138885.857143,\n'
                                             '        "bw_dev" : 275.370200,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 277108,\n'
                                             '        "iops_max" : 278880,\n'
                                             '        "iops_mean" : '
                                             '277771.785714,\n'
                                             '        "iops_stddev" : '
                                             '550.705586,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 36.685714,\n'
                                             '      "sys_cpu" : 61.342857,\n'
                                             '      "ctx" : 11813,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 45.621154,\n'
                                             '        "20" : 52.590952,\n'
                                             '        "50" : 1.747002,\n'
                                             '        "100" : 0.039349,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142899: Wed '
                                             'Oct 26 04:45:17 2022\n'
                                             '  read: IOPS=278k, BW=136MiB/s '
                                             '(142MB/s)(949MiB/7001msec)\n'
                                             '    slat (nsec): min=1472, '
                                             'max=23805, avg=2275.90, '
                                             'stdev=900.41\n'
                                             '    clat (usec): min=4, max=133, '
                                             'avg=11.33, stdev= 2.85\n'
                                             '     lat (usec): min=7, max=136, '
                                             'avg=13.61, stdev= 2.63\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 9408],  '
                                             '5.00th=[ 9536], 10.00th=[ 9536], '
                                             '20.00th=[ 9664],\n'
                                             '     | 30.00th=[ 9664], '
                                             '40.00th=[ 9792], '
                                             '50.00th=[11072], '
                                             '60.00th=[12096],\n'
                                             '     | 70.00th=[12224], '
                                             '80.00th=[12352], '
                                             '90.00th=[12352], '
                                             '95.00th=[12608],\n'
                                             '     | 99.00th=[23680], '
                                             '99.50th=[28288], '
                                             '99.90th=[36096], '
                                             '99.95th=[40192],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=138554, '
                                             'max=139440, per=100.00%, '
                                             'avg=138885.86, stdev=275.37, '
                                             'samples=14\n'
                                             '   iops        : min=277108, '
                                             'max=278880, avg=277771.79, '
                                             'stdev=550.71, samples=14\n'
                                             '  lat (usec)   : 10=45.62%, '
                                             '20=52.59%, 50=1.75%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=36.69%, '
                                             'sys=61.34%, ctx=11813, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1944131,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=136MiB/s (142MB/s), '
                                             '136MiB/s-136MiB/s '
                                             '(142MB/s-142MB/s), io=949MiB '
                                             '(995MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.output'),
                                   'state': {'begin': 1666745227.883581,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8156938552856445,
                                             'end': 1666745228.6992748,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_291.state')},
                       'cmd_292': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.output'),
                                   'state': {'begin': 1666745229.7561324,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8139240741729736,
                                             'end': 1666745230.5700564,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_292.state')},
                       'cmd_293': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.output'),
                                   'state': {'begin': 1666745230.5723808,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.087549924850464,
                                             'end': 1666745241.6599307,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_293.state')},
                       'cmd_294': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752330,\n'
                                             '  "timestamp_ms" : '
                                             '1666752330841,\n'
                                             '  "time" : "Wed Oct 26 04:45:30 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : 998190080,\n'
                                             '        "io_kbytes" : 974795,\n'
                                             '        "bw_bytes" : 142578214,\n'
                                             '        "bw" : 139236,\n'
                                             '        "iops" : 278472.503928,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1949586,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1462,\n'
                                             '          "max" : 34996,\n'
                                             '          "mean" : 2286.772988,\n'
                                             '          "stddev" : '
                                             '911.175884,\n'
                                             '          "N" : 1949587\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 501,\n'
                                             '          "max" : 134612,\n'
                                             '          "mean" : '
                                             '11299.563884,\n'
                                             '          "stddev" : '
                                             '2830.245084,\n'
                                             '          "N" : 1949589,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 9408,\n'
                                             '            "5.000000" : 9536,\n'
                                             '            "10.000000" : 9536,\n'
                                             '            "20.000000" : 9536,\n'
                                             '            "30.000000" : 9664,\n'
                                             '            "40.000000" : 9664,\n'
                                             '            "50.000000" : '
                                             '10944,\n'
                                             '            "60.000000" : '
                                             '12096,\n'
                                             '            "70.000000" : '
                                             '12224,\n'
                                             '            "80.000000" : '
                                             '12224,\n'
                                             '            "90.000000" : '
                                             '12352,\n'
                                             '            "95.000000" : '
                                             '12608,\n'
                                             '            "99.000000" : '
                                             '23680,\n'
                                             '            "99.500000" : '
                                             '28288,\n'
                                             '            "99.900000" : '
                                             '36096,\n'
                                             '            "99.950000" : '
                                             '41216,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 8206,\n'
                                             '          "max" : 138840,\n'
                                             '          "mean" : '
                                             '13586.337374,\n'
                                             '          "stddev" : '
                                             '2615.938007,\n'
                                             '          "N" : 1949589\n'
                                             '        },\n'
                                             '        "bw_min" : 138799,\n'
                                             '        "bw_max" : 139789,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '139298.538462,\n'
                                             '        "bw_dev" : 326.805145,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 277600,\n'
                                             '        "iops_max" : 279578,\n'
                                             '        "iops_mean" : '
                                             '278597.230769,\n'
                                             '        "iops_stddev" : '
                                             '653.355717,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 38.457143,\n'
                                             '      "sys_cpu" : 59.557143,\n'
                                             '      "ctx" : 11881,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 45.654565,\n'
                                             '        "20" : 52.723501,\n'
                                             '        "50" : 1.577668,\n'
                                             '        "100" : 0.043599,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142939: Wed '
                                             'Oct 26 04:45:30 2022\n'
                                             '  read: IOPS=278k, BW=136MiB/s '
                                             '(143MB/s)(952MiB/7001msec)\n'
                                             '    slat (nsec): min=1462, '
                                             'max=34996, avg=2286.77, '
                                             'stdev=911.18\n'
                                             '    clat (nsec): min=501, '
                                             'max=134612, avg=11299.56, '
                                             'stdev=2830.25\n'
                                             '     lat (usec): min=8, max=138, '
                                             'avg=13.59, stdev= 2.62\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 9408],  '
                                             '5.00th=[ 9536], 10.00th=[ 9536], '
                                             '20.00th=[ 9536],\n'
                                             '     | 30.00th=[ 9664], '
                                             '40.00th=[ 9664], '
                                             '50.00th=[10944], '
                                             '60.00th=[12096],\n'
                                             '     | 70.00th=[12224], '
                                             '80.00th=[12224], '
                                             '90.00th=[12352], '
                                             '95.00th=[12608],\n'
                                             '     | 99.00th=[23680], '
                                             '99.50th=[28288], '
                                             '99.90th=[36096], '
                                             '99.95th=[41216],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=138799, '
                                             'max=139789, per=100.00%, '
                                             'avg=139298.54, stdev=326.81, '
                                             'samples=13\n'
                                             '   iops        : min=277600, '
                                             'max=279578, avg=278597.23, '
                                             'stdev=653.36, samples=13\n'
                                             '  lat (nsec)   : 750=0.01%\n'
                                             '  lat (usec)   : 10=45.65%, '
                                             '20=52.72%, 50=1.58%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=38.46%, '
                                             'sys=59.56%, ctx=11881, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1949586,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=136MiB/s (143MB/s), '
                                             '136MiB/s-136MiB/s '
                                             '(143MB/s-143MB/s), io=952MiB '
                                             '(998MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.output'),
                                   'state': {'begin': 1666745241.6631203,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7908663749694824,
                                             'end': 1666745242.4539866,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_294.state')},
                       'cmd_295': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.output'),
                                   'state': {'begin': 1666745243.446813,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7882969379425049,
                                             'end': 1666745244.23511,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_295.state')},
                       'cmd_296': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.output'),
                                   'state': {'begin': 1666745244.2373972,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_4_libaio-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.09430480003357,
                                             'end': 1666745255.331702,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_296.state')},
                       'cmd_297': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752344,\n'
                                             '  "timestamp_ms" : '
                                             '1666752344512,\n'
                                             '  "time" : "Wed Oct 26 04:45:44 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1003818496,\n'
                                             '        "io_kbytes" : 980291,\n'
                                             '        "bw_bytes" : 143382159,\n'
                                             '        "bw" : 140021,\n'
                                             '        "iops" : 280042.708185,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1960579,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1462,\n'
                                             '          "max" : 27762,\n'
                                             '          "mean" : 2373.694871,\n'
                                             '          "stddev" : '
                                             '1047.956777,\n'
                                             '          "N" : 1960580\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4919,\n'
                                             '          "max" : 134282,\n'
                                             '          "mean" : '
                                             '11239.872907,\n'
                                             '          "stddev" : '
                                             '2798.284062,\n'
                                             '          "N" : 1960582,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 9408,\n'
                                             '            "5.000000" : 9408,\n'
                                             '            "10.000000" : 9408,\n'
                                             '            "20.000000" : 9536,\n'
                                             '            "30.000000" : 9536,\n'
                                             '            "40.000000" : 9664,\n'
                                             '            "50.000000" : '
                                             '10944,\n'
                                             '            "60.000000" : '
                                             '11968,\n'
                                             '            "70.000000" : '
                                             '12096,\n'
                                             '            "80.000000" : '
                                             '12224,\n'
                                             '            "90.000000" : '
                                             '12352,\n'
                                             '            "95.000000" : '
                                             '12608,\n'
                                             '            "99.000000" : '
                                             '23680,\n'
                                             '            "99.500000" : '
                                             '28288,\n'
                                             '            "99.900000" : '
                                             '36096,\n'
                                             '            "99.950000" : '
                                             '40192,\n'
                                             '            "99.990000" : 70144\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 7174,\n'
                                             '          "max" : 138400,\n'
                                             '          "mean" : '
                                             '13613.568208,\n'
                                             '          "stddev" : '
                                             '2576.989435,\n'
                                             '          "N" : 1960582\n'
                                             '        },\n'
                                             '        "bw_min" : 139649,\n'
                                             '        "bw_max" : 140398,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '140066.571429,\n'
                                             '        "bw_dev" : 220.341593,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 279298,\n'
                                             '        "iops_max" : 280796,\n'
                                             '        "iops_mean" : '
                                             '280133.214286,\n'
                                             '        "iops_stddev" : '
                                             '440.635935,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 38.142857,\n'
                                             '      "sys_cpu" : 59.885714,\n'
                                             '      "ctx" : 11851,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 45.917150,\n'
                                             '        "20" : 52.538204,\n'
                                             '        "50" : 1.504913,\n'
                                             '        "100" : 0.039019,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=142978: Wed '
                                             'Oct 26 04:45:44 2022\n'
                                             '  read: IOPS=280k, BW=137MiB/s '
                                             '(143MB/s)(957MiB/7001msec)\n'
                                             '    slat (nsec): min=1462, '
                                             'max=27762, avg=2373.69, '
                                             'stdev=1047.96\n'
                                             '    clat (usec): min=4, max=134, '
                                             'avg=11.24, stdev= 2.80\n'
                                             '     lat (usec): min=7, max=138, '
                                             'avg=13.61, stdev= 2.58\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 9408],  '
                                             '5.00th=[ 9408], 10.00th=[ 9408], '
                                             '20.00th=[ 9536],\n'
                                             '     | 30.00th=[ 9536], '
                                             '40.00th=[ 9664], '
                                             '50.00th=[10944], '
                                             '60.00th=[11968],\n'
                                             '     | 70.00th=[12096], '
                                             '80.00th=[12224], '
                                             '90.00th=[12352], '
                                             '95.00th=[12608],\n'
                                             '     | 99.00th=[23680], '
                                             '99.50th=[28288], '
                                             '99.90th=[36096], '
                                             '99.95th=[40192],\n'
                                             '     | 99.99th=[70144]\n'
                                             '   bw (  KiB/s): min=139649, '
                                             'max=140398, per=100.00%, '
                                             'avg=140066.57, stdev=220.34, '
                                             'samples=14\n'
                                             '   iops        : min=279298, '
                                             'max=280796, avg=280133.21, '
                                             'stdev=440.64, samples=14\n'
                                             '  lat (usec)   : 10=45.92%, '
                                             '20=52.54%, 50=1.50%, 100=0.04%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=38.14%, '
                                             'sys=59.89%, ctx=11851, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1960579,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=137MiB/s (143MB/s), '
                                             '137MiB/s-137MiB/s '
                                             '(143MB/s-143MB/s), io=957MiB '
                                             '(1004MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.output'),
                                   'state': {'begin': 1666745255.334885,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_4_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7952744960784912,
                                             'end': 1666745256.1301594,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_297.state')},
                       'cmd_298': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.output'),
                                   'state': {'begin': 1666745257.1318786,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.780454158782959,
                                             'end': 1666745257.9123328,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_298.state')},
                       'cmd_299': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.output'),
                                   'state': {'begin': 1666745257.9144704,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.047147750854492,
                                             'end': 1666745268.9616182,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_299.state')},
                       'cmd_30': {'output': 'io_uring-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751067,\n'
                                            '  "timestamp_ms" : '
                                            '1666751067207,\n'
                                            '  "time" : "Wed Oct 26 04:24:27 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "io_uring",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 282276352,\n'
                                            '        "io_kbytes" : 275660,\n'
                                            '        "bw_bytes" : 40319433,\n'
                                            '        "bw" : 39374,\n'
                                            '        "iops" : 78748.893015,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 551321,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2504,\n'
                                            '          "max" : 49713,\n'
                                            '          "mean" : 2754.340312,\n'
                                            '          "stddev" : 211.568405,\n'
                                            '          "N" : 551321\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 200,\n'
                                            '          "max" : 99306,\n'
                                            '          "mean" : 9273.575117,\n'
                                            '          "stddev" : '
                                            '1277.471589,\n'
                                            '          "N" : 551321,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8896,\n'
                                            '            "5.000000" : 8896,\n'
                                            '            "10.000000" : 9024,\n'
                                            '            "20.000000" : 9024,\n'
                                            '            "30.000000" : 9024,\n'
                                            '            "40.000000" : 9024,\n'
                                            '            "50.000000" : 9152,\n'
                                            '            "60.000000" : 9152,\n'
                                            '            "70.000000" : 9152,\n'
                                            '            "80.000000" : 9152,\n'
                                            '            "90.000000" : 9280,\n'
                                            '            "95.000000" : 9536,\n'
                                            '            "99.000000" : 15168,\n'
                                            '            "99.500000" : 15808,\n'
                                            '            "99.900000" : 19840,\n'
                                            '            "99.950000" : 26240,\n'
                                            '            "99.990000" : 56064\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9197,\n'
                                            '          "max" : 102102,\n'
                                            '          "mean" : 12027.915429,\n'
                                            '          "stddev" : '
                                            '1296.149259,\n'
                                            '          "N" : 551321\n'
                                            '        },\n'
                                            '        "bw_min" : 39234,\n'
                                            '        "bw_max" : 39603,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '39390.142857,\n'
                                            '        "bw_dev" : 106.081142,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 78468,\n'
                                            '        "iops_max" : 79206,\n'
                                            '        "iops_mean" : '
                                            '78780.357143,\n'
                                            '        "iops_stddev" : '
                                            '212.094176,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 12.414286,\n'
                                            '      "sys_cpu" : 48.628571,\n'
                                            '      "ctx" : 551215,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 97.031312,\n'
                                            '        "20" : 2.883982,\n'
                                            '        "50" : 0.057317,\n'
                                            '        "100" : 0.019408,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 793290,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5015,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5015,\n'
                                            '      "util" : 99.049128\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=139058: Wed '
                                            'Oct 26 04:24:27 2022\n'
                                            '  read: IOPS=78.7k, BW=38.5MiB/s '
                                            '(40.3MB/s)(269MiB/7001msec)\n'
                                            '    slat (nsec): min=2504, '
                                            'max=49713, avg=2754.34, '
                                            'stdev=211.57\n'
                                            '    clat (nsec): min=200, '
                                            'max=99306, avg=9273.58, '
                                            'stdev=1277.47\n'
                                            '     lat (usec): min=9, max=102, '
                                            'avg=12.03, stdev= 1.30\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8896],  5.00th=[ '
                                            '8896], 10.00th=[ 9024], 20.00th=[ '
                                            '9024],\n'
                                            '     | 30.00th=[ 9024], 40.00th=[ '
                                            '9024], 50.00th=[ 9152], 60.00th=[ '
                                            '9152],\n'
                                            '     | 70.00th=[ 9152], 80.00th=[ '
                                            '9152], 90.00th=[ 9280], 95.00th=[ '
                                            '9536],\n'
                                            '     | 99.00th=[15168], '
                                            '99.50th=[15808], 99.90th=[19840], '
                                            '99.95th=[26240],\n'
                                            '     | 99.99th=[56064]\n'
                                            '   bw (  KiB/s): min=39234, '
                                            'max=39603, per=100.00%, '
                                            'avg=39390.14, stdev=106.08, '
                                            'samples=14\n'
                                            '   iops        : min=78468, '
                                            'max=79206, avg=78780.36, '
                                            'stdev=212.09, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 10=97.03%, '
                                            '20=2.88%, 50=0.06%, 100=0.02%\n'
                                            '  cpu          : usr=12.41%, '
                                            'sys=48.63%, ctx=551215, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=551321,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=38.5MiB/s (40.3MB/s), '
                                            '38.5MiB/s-38.5MiB/s '
                                            '(40.3MB/s-40.3MB/s), io=269MiB '
                                            '(282MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=793290/0, '
                                            'merge=0/0, ticks=5015/0, '
                                            'in_queue=5015, util=99.05%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.output'),
                                  'state': {'begin': 1666743978.008007,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7910122871398926,
                                            'end': 1666743978.7990193,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_30.state')},
                       'cmd_300': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752358,\n'
                                             '  "timestamp_ms" : '
                                             '1666752358142,\n'
                                             '  "time" : "Wed Oct 26 04:45:58 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1195644416,\n'
                                             '        "io_kbytes" : 1167621,\n'
                                             '        "bw_bytes" : 170781947,\n'
                                             '        "bw" : 166779,\n'
                                             '        "iops" : 333557.491787,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2335236,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1152,\n'
                                             '          "max" : 19917,\n'
                                             '          "mean" : 2244.668577,\n'
                                             '          "stddev" : '
                                             '1068.788020,\n'
                                             '          "N" : 2335236\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 80,\n'
                                             '          "max" : 145854,\n'
                                             '          "mean" : '
                                             '21249.490787,\n'
                                             '          "stddev" : '
                                             '3863.814320,\n'
                                             '          "N" : 2335243,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13760,\n'
                                             '            "5.000000" : 18816,\n'
                                             '            "10.000000" : '
                                             '19072,\n'
                                             '            "20.000000" : '
                                             '19328,\n'
                                             '            "30.000000" : '
                                             '20352,\n'
                                             '            "40.000000" : '
                                             '20608,\n'
                                             '            "50.000000" : '
                                             '20608,\n'
                                             '            "60.000000" : '
                                             '20864,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21888,\n'
                                             '            "90.000000" : '
                                             '22656,\n'
                                             '            "95.000000" : '
                                             '24960,\n'
                                             '            "99.000000" : '
                                             '41728,\n'
                                             '            "99.500000" : '
                                             '44288,\n'
                                             '            "99.900000" : '
                                             '56064,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 82432\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6612,\n'
                                             '          "max" : 147096,\n'
                                             '          "mean" : '
                                             '23494.157402,\n'
                                             '          "stddev" : '
                                             '3821.091989,\n'
                                             '          "N" : 2335243\n'
                                             '        },\n'
                                             '        "bw_min" : 166104,\n'
                                             '        "bw_max" : 167290,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '166794.153846,\n'
                                             '        "bw_dev" : 286.537329,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 332210,\n'
                                             '        "iops_max" : 334580,\n'
                                             '        "iops_mean" : '
                                             '333588.461538,\n'
                                             '        "iops_stddev" : '
                                             '572.567989,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.142857,\n'
                                             '      "sys_cpu" : 83.414286,\n'
                                             '      "ctx" : 12659,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999957,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.016915,\n'
                                             '        "20" : 22.109457,\n'
                                             '        "50" : 77.740494,\n'
                                             '        "100" : 0.130993,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=143018: Wed Oct 26 04:45:58 '
                                             '2022\n'
                                             '  read: IOPS=334k, BW=163MiB/s '
                                             '(171MB/s)(1140MiB/7001msec)\n'
                                             '    slat (nsec): min=1152, '
                                             'max=19917, avg=2244.67, '
                                             'stdev=1068.79\n'
                                             '    clat (nsec): min=80, '
                                             'max=145854, avg=21249.49, '
                                             'stdev=3863.81\n'
                                             '     lat (usec): min=6, max=147, '
                                             'avg=23.49, stdev= 3.82\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13760],  '
                                             '5.00th=[18816], 10.00th=[19072], '
                                             '20.00th=[19328],\n'
                                             '     | 30.00th=[20352], '
                                             '40.00th=[20608], '
                                             '50.00th=[20608], '
                                             '60.00th=[20864],\n'
                                             '     | 70.00th=[21376], '
                                             '80.00th=[21888], '
                                             '90.00th=[22656], '
                                             '95.00th=[24960],\n'
                                             '     | 99.00th=[41728], '
                                             '99.50th=[44288], '
                                             '99.90th=[56064], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[82432]\n'
                                             '   bw (  KiB/s): min=166104, '
                                             'max=167290, per=100.00%, '
                                             'avg=166794.15, stdev=286.54, '
                                             'samples=13\n'
                                             '   iops        : min=332210, '
                                             'max=334580, avg=333588.46, '
                                             'stdev=572.57, samples=13\n'
                                             '  lat (nsec)   : 100=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.02%, 20=22.11%, 50=77.74%, '
                                             '100=0.13%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=14.14%, '
                                             'sys=83.41%, ctx=12659, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2335236,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=163MiB/s (171MB/s), '
                                             '163MiB/s-163MiB/s '
                                             '(171MB/s-171MB/s), io=1140MiB '
                                             '(1196MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.output'),
                                   'state': {'begin': 1666745268.964636,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7934114933013916,
                                             'end': 1666745269.7580476,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_300.state')},
                       'cmd_301': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.output'),
                                   'state': {'begin': 1666745270.7479367,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.781456470489502,
                                             'end': 1666745271.5293932,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_301.state')},
                       'cmd_302': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.output'),
                                   'state': {'begin': 1666745271.5315776,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.040171384811401,
                                             'end': 1666745282.571749,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_302.state')},
                       'cmd_303': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752371,\n'
                                             '  "timestamp_ms" : '
                                             '1666752371752,\n'
                                             '  "time" : "Wed Oct 26 04:46:11 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1196563968,\n'
                                             '        "io_kbytes" : 1168519,\n'
                                             '        "bw_bytes" : 170913293,\n'
                                             '        "bw" : 166907,\n'
                                             '        "iops" : 333813.883731,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2337031,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1152,\n'
                                             '          "max" : 24005,\n'
                                             '          "mean" : 2220.236975,\n'
                                             '          "stddev" : '
                                             '1090.837490,\n'
                                             '          "N" : 2337032\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 3196,\n'
                                             '          "max" : 144360,\n'
                                             '          "mean" : '
                                             '21244.316029,\n'
                                             '          "stddev" : '
                                             '3839.831790,\n'
                                             '          "N" : 2337038,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13888,\n'
                                             '            "5.000000" : 18816,\n'
                                             '            "10.000000" : '
                                             '19072,\n'
                                             '            "20.000000" : '
                                             '19328,\n'
                                             '            "30.000000" : '
                                             '20352,\n'
                                             '            "40.000000" : '
                                             '20608,\n'
                                             '            "50.000000" : '
                                             '20608,\n'
                                             '            "60.000000" : '
                                             '20864,\n'
                                             '            "70.000000" : '
                                             '21632,\n'
                                             '            "80.000000" : '
                                             '21888,\n'
                                             '            "90.000000" : '
                                             '22912,\n'
                                             '            "95.000000" : '
                                             '24960,\n'
                                             '            "99.000000" : '
                                             '41728,\n'
                                             '            "99.500000" : '
                                             '44288,\n'
                                             '            "99.900000" : '
                                             '53504,\n'
                                             '            "99.950000" : '
                                             '64768,\n'
                                             '            "99.990000" : 81408\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 7093,\n'
                                             '          "max" : 145563,\n'
                                             '          "mean" : '
                                             '23464.552185,\n'
                                             '          "stddev" : '
                                             '3807.683431,\n'
                                             '          "N" : 2337038\n'
                                             '        },\n'
                                             '        "bw_min" : 166713,\n'
                                             '        "bw_max" : 167275,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '166953.357143,\n'
                                             '        "bw_dev" : 172.887517,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 333426,\n'
                                             '        "iops_max" : 334550,\n'
                                             '        "iops_mean" : '
                                             '333906.928571,\n'
                                             '        "iops_stddev" : '
                                             '345.829763,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 13.800000,\n'
                                             '      "sys_cpu" : 83.771429,\n'
                                             '      "ctx" : 12624,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999957,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.015575,\n'
                                             '        "20" : 23.539739,\n'
                                             '        "50" : 76.322522,\n'
                                             '        "100" : 0.120965,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=143058: Wed Oct 26 04:46:11 '
                                             '2022\n'
                                             '  read: IOPS=334k, BW=163MiB/s '
                                             '(171MB/s)(1141MiB/7001msec)\n'
                                             '    slat (nsec): min=1152, '
                                             'max=24005, avg=2220.24, '
                                             'stdev=1090.84\n'
                                             '    clat (usec): min=3, max=144, '
                                             'avg=21.24, stdev= 3.84\n'
                                             '     lat (usec): min=7, max=145, '
                                             'avg=23.46, stdev= 3.81\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13888],  '
                                             '5.00th=[18816], 10.00th=[19072], '
                                             '20.00th=[19328],\n'
                                             '     | 30.00th=[20352], '
                                             '40.00th=[20608], '
                                             '50.00th=[20608], '
                                             '60.00th=[20864],\n'
                                             '     | 70.00th=[21632], '
                                             '80.00th=[21888], '
                                             '90.00th=[22912], '
                                             '95.00th=[24960],\n'
                                             '     | 99.00th=[41728], '
                                             '99.50th=[44288], '
                                             '99.90th=[53504], '
                                             '99.95th=[64768],\n'
                                             '     | 99.99th=[81408]\n'
                                             '   bw (  KiB/s): min=166713, '
                                             'max=167275, per=100.00%, '
                                             'avg=166953.36, stdev=172.89, '
                                             'samples=14\n'
                                             '   iops        : min=333426, '
                                             'max=334550, avg=333906.93, '
                                             'stdev=345.83, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.02%, 20=23.54%, 50=76.32%, '
                                             '100=0.12%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=13.80%, '
                                             'sys=83.77%, ctx=12624, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2337031,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=163MiB/s (171MB/s), '
                                             '163MiB/s-163MiB/s '
                                             '(171MB/s-171MB/s), io=1141MiB '
                                             '(1197MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.output'),
                                   'state': {'begin': 1666745282.574461,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7908134460449219,
                                             'end': 1666745283.3652744,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_303.state')},
                       'cmd_304': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.output'),
                                   'state': {'begin': 1666745284.3566759,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.779430627822876,
                                             'end': 1666745285.1361065,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_304.state')},
                       'cmd_305': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.output'),
                                   'state': {'begin': 1666745285.1381803,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.040968179702759,
                                             'end': 1666745296.1791484,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_305.state')},
                       'cmd_306': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752385,\n'
                                             '  "timestamp_ms" : '
                                             '1666752385359,\n'
                                             '  "time" : "Wed Oct 26 04:46:25 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1210347520,\n'
                                             '        "io_kbytes" : 1181980,\n'
                                             '        "bw_bytes" : 172882091,\n'
                                             '        "bw" : 168830,\n'
                                             '        "iops" : 337659.334381,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2363953,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1122,\n'
                                             '          "max" : 22653,\n'
                                             '          "mean" : 2034.197053,\n'
                                             '          "stddev" : '
                                             '1011.133446,\n'
                                             '          "N" : 2363953\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 90,\n'
                                             '          "max" : 142347,\n'
                                             '          "mean" : '
                                             '21168.681512,\n'
                                             '          "stddev" : '
                                             '3823.819840,\n'
                                             '          "N" : 2363960,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14400,\n'
                                             '            "5.000000" : 18560,\n'
                                             '            "10.000000" : '
                                             '18816,\n'
                                             '            "20.000000" : '
                                             '19072,\n'
                                             '            "30.000000" : '
                                             '20352,\n'
                                             '            "40.000000" : '
                                             '20352,\n'
                                             '            "50.000000" : '
                                             '20608,\n'
                                             '            "60.000000" : '
                                             '21120,\n'
                                             '            "70.000000" : '
                                             '21632,\n'
                                             '            "80.000000" : '
                                             '21888,\n'
                                             '            "90.000000" : '
                                             '22912,\n'
                                             '            "95.000000" : '
                                             '24704,\n'
                                             '            "99.000000" : '
                                             '41216,\n'
                                             '            "99.500000" : '
                                             '43776,\n'
                                             '            "99.900000" : '
                                             '55040,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 80384\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5551,\n'
                                             '          "max" : 143539,\n'
                                             '          "mean" : '
                                             '23202.878542,\n'
                                             '          "stddev" : '
                                             '3787.397119,\n'
                                             '          "N" : 2363960\n'
                                             '        },\n'
                                             '        "bw_min" : 168266,\n'
                                             '        "bw_max" : 169275,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '168882.153846,\n'
                                             '        "bw_dev" : 336.167579,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 336532,\n'
                                             '        "iops_max" : 338550,\n'
                                             '        "iops_mean" : '
                                             '337764.307692,\n'
                                             '        "iops_stddev" : '
                                             '672.444221,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 13.700000,\n'
                                             '      "sys_cpu" : 83.814286,\n'
                                             '      "ctx" : 12774,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999958,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.013790,\n'
                                             '        "20" : 23.539004,\n'
                                             '        "50" : 76.321145,\n'
                                             '        "100" : 0.124791,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=143096: Wed Oct 26 04:46:25 '
                                             '2022\n'
                                             '  read: IOPS=338k, BW=165MiB/s '
                                             '(173MB/s)(1154MiB/7001msec)\n'
                                             '    slat (nsec): min=1122, '
                                             'max=22653, avg=2034.20, '
                                             'stdev=1011.13\n'
                                             '    clat (nsec): min=90, '
                                             'max=142347, avg=21168.68, '
                                             'stdev=3823.82\n'
                                             '     lat (usec): min=5, max=143, '
                                             'avg=23.20, stdev= 3.79\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[14400],  '
                                             '5.00th=[18560], 10.00th=[18816], '
                                             '20.00th=[19072],\n'
                                             '     | 30.00th=[20352], '
                                             '40.00th=[20352], '
                                             '50.00th=[20608], '
                                             '60.00th=[21120],\n'
                                             '     | 70.00th=[21632], '
                                             '80.00th=[21888], '
                                             '90.00th=[22912], '
                                             '95.00th=[24704],\n'
                                             '     | 99.00th=[41216], '
                                             '99.50th=[43776], '
                                             '99.90th=[55040], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[80384]\n'
                                             '   bw (  KiB/s): min=168266, '
                                             'max=169275, per=100.00%, '
                                             'avg=168882.15, stdev=336.17, '
                                             'samples=13\n'
                                             '   iops        : min=336532, '
                                             'max=338550, avg=337764.31, '
                                             'stdev=672.44, samples=13\n'
                                             '  lat (nsec)   : 100=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '10=0.01%, 20=23.54%, 50=76.32%, '
                                             '100=0.12%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=13.70%, '
                                             'sys=83.81%, ctx=12774, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2363953,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=165MiB/s (173MB/s), '
                                             '165MiB/s-165MiB/s '
                                             '(173MB/s-173MB/s), io=1154MiB '
                                             '(1210MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.output'),
                                   'state': {'begin': 1666745296.182085,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7856402397155762,
                                             'end': 1666745296.9677253,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_306.state')},
                       'cmd_307': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.output'),
                                   'state': {'begin': 1666745297.9606037,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7840566635131836,
                                             'end': 1666745298.7446604,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_307.state')},
                       'cmd_308': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.output'),
                                   'state': {'begin': 1666745298.7467673,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.065226793289185,
                                             'end': 1666745309.811994,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_308.state')},
                       'cmd_309': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752398,\n'
                                             '  "timestamp_ms" : '
                                             '1666752398992,\n'
                                             '  "time" : "Wed Oct 26 04:46:38 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1159592448,\n'
                                             '        "io_kbytes" : 1132414,\n'
                                             '        "bw_bytes" : 165632402,\n'
                                             '        "bw" : 161750,\n'
                                             '        "iops" : 323499.642908,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2264821,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1212,\n'
                                             '          "max" : 32892,\n'
                                             '          "mean" : 2529.842174,\n'
                                             '          "stddev" : '
                                             '737.661078,\n'
                                             '          "N" : 2264822\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 120,\n'
                                             '          "max" : 143188,\n'
                                             '          "mean" : '
                                             '21923.724910,\n'
                                             '          "stddev" : '
                                             '3812.563652,\n'
                                             '          "N" : 2264828,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14272,\n'
                                             '            "5.000000" : 19840,\n'
                                             '            "10.000000" : '
                                             '20352,\n'
                                             '            "20.000000" : '
                                             '20864,\n'
                                             '            "30.000000" : '
                                             '21120,\n'
                                             '            "40.000000" : '
                                             '21120,\n'
                                             '            "50.000000" : '
                                             '21376,\n'
                                             '            "60.000000" : '
                                             '21376,\n'
                                             '            "70.000000" : '
                                             '21632,\n'
                                             '            "80.000000" : '
                                             '21888,\n'
                                             '            "90.000000" : '
                                             '22912,\n'
                                             '            "95.000000" : '
                                             '25984,\n'
                                             '            "99.000000" : '
                                             '42240,\n'
                                             '            "99.500000" : '
                                             '44800,\n'
                                             '            "99.900000" : '
                                             '55040,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 81408\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6672,\n'
                                             '          "max" : 144431,\n'
                                             '          "mean" : '
                                             '24453.567314,\n'
                                             '          "stddev" : '
                                             '3663.939270,\n'
                                             '          "N" : 2264828\n'
                                             '        },\n'
                                             '        "bw_min" : 161462,\n'
                                             '        "bw_max" : 162202,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '161798.928571,\n'
                                             '        "bw_dev" : 238.197870,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 322924,\n'
                                             '        "iops_max" : 324404,\n'
                                             '        "iops_mean" : '
                                             '323597.928571,\n'
                                             '        "iops_stddev" : '
                                             '476.383889,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.985714,\n'
                                             '      "sys_cpu" : 81.714286,\n'
                                             '      "ctx" : 12576,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999978,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.018721,\n'
                                             '        "20" : 6.321383,\n'
                                             '        "50" : 93.517589,\n'
                                             '        "100" : 0.139349,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=143134: Wed '
                                             'Oct 26 04:46:38 2022\n'
                                             '  read: IOPS=323k, BW=158MiB/s '
                                             '(166MB/s)(1106MiB/7001msec)\n'
                                             '    slat (nsec): min=1212, '
                                             'max=32892, avg=2529.84, '
                                             'stdev=737.66\n'
                                             '    clat (nsec): min=120, '
                                             'max=143188, avg=21923.72, '
                                             'stdev=3812.56\n'
                                             '     lat (usec): min=6, max=144, '
                                             'avg=24.45, stdev= 3.66\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[14272],  '
                                             '5.00th=[19840], 10.00th=[20352], '
                                             '20.00th=[20864],\n'
                                             '     | 30.00th=[21120], '
                                             '40.00th=[21120], '
                                             '50.00th=[21376], '
                                             '60.00th=[21376],\n'
                                             '     | 70.00th=[21632], '
                                             '80.00th=[21888], '
                                             '90.00th=[22912], '
                                             '95.00th=[25984],\n'
                                             '     | 99.00th=[42240], '
                                             '99.50th=[44800], '
                                             '99.90th=[55040], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[81408]\n'
                                             '   bw (  KiB/s): min=161462, '
                                             'max=162202, per=100.00%, '
                                             'avg=161798.93, stdev=238.20, '
                                             'samples=14\n'
                                             '   iops        : min=322924, '
                                             'max=324404, avg=323597.93, '
                                             'stdev=476.38, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.02%, 20=6.32%, '
                                             '50=93.52%\n'
                                             '  lat (usec)   : 100=0.14%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=15.99%, '
                                             'sys=81.71%, ctx=12576, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2264821,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=158MiB/s (166MB/s), '
                                             '158MiB/s-158MiB/s '
                                             '(166MB/s-166MB/s), io=1106MiB '
                                             '(1160MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.output'),
                                   'state': {'begin': 1666745309.8151252,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7887105941772461,
                                             'end': 1666745310.6038358,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_309.state')},
                       'cmd_31': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_512_1_io_uring-reference_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.output'),
                                  'state': {'begin': 1666743979.7884958,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7848758697509766,
                                            'end': 1666743980.5733716,
                                            'err': 1,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_31.state')},
                       'cmd_310': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.output'),
                                   'state': {'begin': 1666745311.588529,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7839469909667969,
                                             'end': 1666745312.372476,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_310.state')},
                       'cmd_311': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.output'),
                                   'state': {'begin': 1666745312.3744292,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.052843809127808,
                                             'end': 1666745323.427273,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_311.state')},
                       'cmd_312': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752412,\n'
                                             '  "timestamp_ms" : '
                                             '1666752412607,\n'
                                             '  "time" : "Wed Oct 26 04:46:52 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1175407104,\n'
                                             '        "io_kbytes" : 1147858,\n'
                                             '        "bw_bytes" : 167891316,\n'
                                             '        "bw" : 163956,\n'
                                             '        "iops" : 327911.584059,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2295709,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1202,\n'
                                             '          "max" : 46768,\n'
                                             '          "mean" : 2507.877895,\n'
                                             '          "stddev" : '
                                             '718.916471,\n'
                                             '          "N" : 2295710\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 120,\n'
                                             '          "max" : 143369,\n'
                                             '          "mean" : '
                                             '21623.022268,\n'
                                             '          "stddev" : '
                                             '3822.739674,\n'
                                             '          "N" : 2295716,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14144,\n'
                                             '            "5.000000" : 19840,\n'
                                             '            "10.000000" : '
                                             '20608,\n'
                                             '            "20.000000" : '
                                             '20608,\n'
                                             '            "30.000000" : '
                                             '20864,\n'
                                             '            "40.000000" : '
                                             '20864,\n'
                                             '            "50.000000" : '
                                             '21120,\n'
                                             '            "60.000000" : '
                                             '21120,\n'
                                             '            "70.000000" : '
                                             '21120,\n'
                                             '            "80.000000" : '
                                             '21376,\n'
                                             '            "90.000000" : '
                                             '22656,\n'
                                             '            "95.000000" : '
                                             '25728,\n'
                                             '            "99.000000" : '
                                             '42240,\n'
                                             '            "99.500000" : '
                                             '44288,\n'
                                             '            "99.900000" : '
                                             '56064,\n'
                                             '            "99.950000" : '
                                             '68096,\n'
                                             '            "99.990000" : 83456\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6823,\n'
                                             '          "max" : 144632,\n'
                                             '          "mean" : '
                                             '24130.900482,\n'
                                             '          "stddev" : '
                                             '3672.577138,\n'
                                             '          "N" : 2295716\n'
                                             '        },\n'
                                             '        "bw_min" : 163612,\n'
                                             '        "bw_max" : 164316,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '164015.000000,\n'
                                             '        "bw_dev" : 228.698856,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 327224,\n'
                                             '        "iops_max" : 328633,\n'
                                             '        "iops_mean" : '
                                             '328030.230769,\n'
                                             '        "iops_stddev" : '
                                             '457.340893,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.928571,\n'
                                             '      "sys_cpu" : 81.728571,\n'
                                             '      "ctx" : 12743,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999978,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.016378,\n'
                                             '        "20" : 6.622529,\n'
                                             '        "50" : 93.218435,\n'
                                             '        "100" : 0.138563,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=143172: Wed '
                                             'Oct 26 04:46:52 2022\n'
                                             '  read: IOPS=328k, BW=160MiB/s '
                                             '(168MB/s)(1121MiB/7001msec)\n'
                                             '    slat (nsec): min=1202, '
                                             'max=46768, avg=2507.88, '
                                             'stdev=718.92\n'
                                             '    clat (nsec): min=120, '
                                             'max=143369, avg=21623.02, '
                                             'stdev=3822.74\n'
                                             '     lat (usec): min=6, max=144, '
                                             'avg=24.13, stdev= 3.67\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[14144],  '
                                             '5.00th=[19840], 10.00th=[20608], '
                                             '20.00th=[20608],\n'
                                             '     | 30.00th=[20864], '
                                             '40.00th=[20864], '
                                             '50.00th=[21120], '
                                             '60.00th=[21120],\n'
                                             '     | 70.00th=[21120], '
                                             '80.00th=[21376], '
                                             '90.00th=[22656], '
                                             '95.00th=[25728],\n'
                                             '     | 99.00th=[42240], '
                                             '99.50th=[44288], '
                                             '99.90th=[56064], '
                                             '99.95th=[68096],\n'
                                             '     | 99.99th=[83456]\n'
                                             '   bw (  KiB/s): min=163612, '
                                             'max=164316, per=100.00%, '
                                             'avg=164015.00, stdev=228.70, '
                                             'samples=13\n'
                                             '   iops        : min=327224, '
                                             'max=328633, avg=328030.23, '
                                             'stdev=457.34, samples=13\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.02%, 20=6.62%, '
                                             '50=93.22%\n'
                                             '  lat (usec)   : 100=0.14%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=15.93%, '
                                             'sys=81.73%, ctx=12743, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2295709,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=160MiB/s (168MB/s), '
                                             '160MiB/s-160MiB/s '
                                             '(168MB/s-168MB/s), io=1121MiB '
                                             '(1175MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.output'),
                                   'state': {'begin': 1666745323.4303358,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7909913063049316,
                                             'end': 1666745324.221327,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_312.state')},
                       'cmd_313': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.output'),
                                   'state': {'begin': 1666745325.2078052,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7832224369049072,
                                             'end': 1666745325.9910276,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_313.state')},
                       'cmd_314': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.output'),
                                   'state': {'begin': 1666745325.9934478,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.061598300933838,
                                             'end': 1666745337.055046,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_314.state')},
                       'cmd_315': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752426,\n'
                                             '  "timestamp_ms" : '
                                             '1666752426234,\n'
                                             '  "time" : "Wed Oct 26 04:47:06 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1161004544,\n'
                                             '        "io_kbytes" : 1133793,\n'
                                             '        "bw_bytes" : 165834101,\n'
                                             '        "bw" : 161947,\n'
                                             '        "iops" : 323893.586630,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2267579,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1202,\n'
                                             '          "max" : 38503,\n'
                                             '          "mean" : 2508.514283,\n'
                                             '          "stddev" : '
                                             '748.390671,\n'
                                             '          "N" : 2267580\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 121,\n'
                                             '          "max" : 141366,\n'
                                             '          "mean" : '
                                             '21924.472038,\n'
                                             '          "stddev" : '
                                             '3826.780599,\n'
                                             '          "N" : 2267586,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14272,\n'
                                             '            "5.000000" : 19840,\n'
                                             '            "10.000000" : '
                                             '20352,\n'
                                             '            "20.000000" : '
                                             '20864,\n'
                                             '            "30.000000" : '
                                             '21120,\n'
                                             '            "40.000000" : '
                                             '21120,\n'
                                             '            "50.000000" : '
                                             '21376,\n'
                                             '            "60.000000" : '
                                             '21376,\n'
                                             '            "70.000000" : '
                                             '21632,\n'
                                             '            "80.000000" : '
                                             '22144,\n'
                                             '            "90.000000" : '
                                             '22912,\n'
                                             '            "95.000000" : '
                                             '25728,\n'
                                             '            "99.000000" : '
                                             '42752,\n'
                                             '            "99.500000" : '
                                             '44800,\n'
                                             '            "99.900000" : '
                                             '56576,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 81408\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6502,\n'
                                             '          "max" : 142618,\n'
                                             '          "mean" : '
                                             '24432.986819,\n'
                                             '          "stddev" : '
                                             '3677.160613,\n'
                                             '          "N" : 2267586\n'
                                             '        },\n'
                                             '        "bw_min" : 161484,\n'
                                             '        "bw_max" : 162868,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '161985.571429,\n'
                                             '        "bw_dev" : 433.250099,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 322966,\n'
                                             '        "iops_max" : 325736,\n'
                                             '        "iops_mean" : '
                                             '323971.214286,\n'
                                             '        "iops_stddev" : '
                                             '866.505997,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.057143,\n'
                                             '      "sys_cpu" : 80.628571,\n'
                                             '      "ctx" : 12620,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999978,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.018875,\n'
                                             '        "20" : 6.020430,\n'
                                             '        "50" : 93.815871,\n'
                                             '        "100" : 0.142884,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=143210: Wed '
                                             'Oct 26 04:47:06 2022\n'
                                             '  read: IOPS=324k, BW=158MiB/s '
                                             '(166MB/s)(1107MiB/7001msec)\n'
                                             '    slat (nsec): min=1202, '
                                             'max=38503, avg=2508.51, '
                                             'stdev=748.39\n'
                                             '    clat (nsec): min=121, '
                                             'max=141366, avg=21924.47, '
                                             'stdev=3826.78\n'
                                             '     lat (usec): min=6, max=142, '
                                             'avg=24.43, stdev= 3.68\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[14272],  '
                                             '5.00th=[19840], 10.00th=[20352], '
                                             '20.00th=[20864],\n'
                                             '     | 30.00th=[21120], '
                                             '40.00th=[21120], '
                                             '50.00th=[21376], '
                                             '60.00th=[21376],\n'
                                             '     | 70.00th=[21632], '
                                             '80.00th=[22144], '
                                             '90.00th=[22912], '
                                             '95.00th=[25728],\n'
                                             '     | 99.00th=[42752], '
                                             '99.50th=[44800], '
                                             '99.90th=[56576], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[81408]\n'
                                             '   bw (  KiB/s): min=161484, '
                                             'max=162868, per=100.00%, '
                                             'avg=161985.57, stdev=433.25, '
                                             'samples=14\n'
                                             '   iops        : min=322966, '
                                             'max=325736, avg=323971.21, '
                                             'stdev=866.51, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.02%, 20=6.02%, '
                                             '50=93.82%\n'
                                             '  lat (usec)   : 100=0.14%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=17.06%, '
                                             'sys=80.63%, ctx=12620, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2267579,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=158MiB/s (166MB/s), '
                                             '158MiB/s-158MiB/s '
                                             '(166MB/s-166MB/s), io=1107MiB '
                                             '(1161MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.output'),
                                   'state': {'begin': 1666745337.0583837,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8026723861694336,
                                             'end': 1666745337.861056,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_315.state')},
                       'cmd_316': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.output'),
                                   'state': {'begin': 1666745339.881891,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7861189842224121,
                                             'end': 1666745340.66801,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_316.state')},
                       'cmd_317': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.output'),
                                   'state': {'begin': 1666745340.6700892,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.4807448387146,
                                             'end': 1666745352.150834,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_317.state')},
                       'cmd_318': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752441,\n'
                                             '  "timestamp_ms" : '
                                             '1666752441259,\n'
                                             '  "time" : "Wed Oct 26 04:47:21 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2276859392,\n'
                                             '        "io_kbytes" : 2223495,\n'
                                             '        "bw_bytes" : 325219167,\n'
                                             '        "bw" : 317596,\n'
                                             '        "iops" : 635192.543922,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 4446983,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 190,\n'
                                             '          "max" : 30808,\n'
                                             '          "mean" : 266.992913,\n'
                                             '          "stddev" : 64.891956,\n'
                                             '          "N" : 4446984\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 441,\n'
                                             '          "max" : 133430,\n'
                                             '          "mean" : '
                                             '11928.421873,\n'
                                             '          "stddev" : '
                                             '4431.847865,\n'
                                             '          "N" : 4446983,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7008,\n'
                                             '            "5.000000" : 7904,\n'
                                             '            "10.000000" : 8512,\n'
                                             '            "20.000000" : 9536,\n'
                                             '            "30.000000" : '
                                             '10048,\n'
                                             '            "40.000000" : '
                                             '10560,\n'
                                             '            "50.000000" : '
                                             '10944,\n'
                                             '            "60.000000" : '
                                             '11584,\n'
                                             '            "70.000000" : '
                                             '12224,\n'
                                             '            "80.000000" : '
                                             '13120,\n'
                                             '            "90.000000" : '
                                             '14784,\n'
                                             '            "95.000000" : '
                                             '19840,\n'
                                             '            "99.000000" : '
                                             '31104,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '47872,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5991,\n'
                                             '          "max" : 133711,\n'
                                             '          "mean" : '
                                             '12195.414785,\n'
                                             '          "stddev" : '
                                             '4429.865740,\n'
                                             '          "N" : 4446983\n'
                                             '        },\n'
                                             '        "bw_min" : 317209,\n'
                                             '        "bw_max" : 318994,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '317838.615385,\n'
                                             '        "bw_dev" : 507.818461,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 634418,\n'
                                             '        "iops_max" : 637988,\n'
                                             '        "iops_mean" : '
                                             '635677.538462,\n'
                                             '        "iops_stddev" : '
                                             '1015.420079,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 22,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 24.005084,\n'
                                             '        "8" : 75.986191,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.992103,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 28.089853,\n'
                                             '        "20" : 66.951931,\n'
                                             '        "50" : 4.864332,\n'
                                             '        "100" : 0.091613,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=143259: Wed Oct 26 04:47:21 '
                                             '2022\n'
                                             '  read: IOPS=635k, BW=310MiB/s '
                                             '(325MB/s)(2171MiB/7001msec)\n'
                                             '    slat (nsec): min=190, '
                                             'max=30808, avg=266.99, '
                                             'stdev=64.89\n'
                                             '    clat (nsec): min=441, '
                                             'max=133430, avg=11928.42, '
                                             'stdev=4431.85\n'
                                             '     lat (usec): min=5, max=133, '
                                             'avg=12.20, stdev= 4.43\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7008],  '
                                             '5.00th=[ 7904], 10.00th=[ 8512], '
                                             '20.00th=[ 9536],\n'
                                             '     | 30.00th=[10048], '
                                             '40.00th=[10560], '
                                             '50.00th=[10944], '
                                             '60.00th=[11584],\n'
                                             '     | 70.00th=[12224], '
                                             '80.00th=[13120], '
                                             '90.00th=[14784], '
                                             '95.00th=[19840],\n'
                                             '     | 99.00th=[31104], '
                                             '99.50th=[34560], '
                                             '99.90th=[47872], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=317209, '
                                             'max=318994, per=100.00%, '
                                             'avg=317838.62, stdev=507.82, '
                                             'samples=13\n'
                                             '   iops        : min=634418, '
                                             'max=637988, avg=635677.54, '
                                             'stdev=1015.42, samples=13\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=28.09%, 20=66.95%, '
                                             '50=4.86%\n'
                                             '  lat (usec)   : 100=0.09%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=22, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=24.0%, 8=76.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=4446983,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=310MiB/s (325MB/s), '
                                             '310MiB/s-310MiB/s '
                                             '(325MB/s-325MB/s), io=2171MiB '
                                             '(2277MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.output'),
                                   'state': {'begin': 1666745352.1577332,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7898359298706055,
                                             'end': 1666745352.9475691,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_318.state')},
                       'cmd_319': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.output'),
                                   'state': {'begin': 1666745354.972949,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7823724746704102,
                                             'end': 1666745355.7553215,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_319.state')},
                       'cmd_32': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.output'),
                                  'state': {'begin': 1666743980.5754085,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-reference_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.044062614440918,
                                            'end': 1666743991.619471,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_32.state')},
                       'cmd_320': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.output'),
                                   'state': {'begin': 1666745355.7576373,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.48052453994751,
                                             'end': 1666745367.2381618,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_320.state')},
                       'cmd_321': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752456,\n'
                                             '  "timestamp_ms" : '
                                             '1666752456347,\n'
                                             '  "time" : "Wed Oct 26 04:47:36 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1845574656,\n'
                                             '        "io_kbytes" : 1802319,\n'
                                             '        "bw_bytes" : 263615862,\n'
                                             '        "bw" : 257437,\n'
                                             '        "iops" : 514873.589487,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 3604630,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 170,\n'
                                             '          "max" : 9297,\n'
                                             '          "mean" : 249.728569,\n'
                                             '          "stddev" : 28.581610,\n'
                                             '          "N" : 3604631\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4809,\n'
                                             '          "max" : 130585,\n'
                                             '          "mean" : '
                                             '14363.971523,\n'
                                             '          "stddev" : '
                                             '5215.015737,\n'
                                             '          "N" : 3604630,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 7456,\n'
                                             '            "5.000000" : 8384,\n'
                                             '            "10.000000" : 8896,\n'
                                             '            "20.000000" : 9792,\n'
                                             '            "30.000000" : '
                                             '10560,\n'
                                             '            "40.000000" : '
                                             '11712,\n'
                                             '            "50.000000" : '
                                             '14144,\n'
                                             '            "60.000000" : '
                                             '15424,\n'
                                             '            "70.000000" : '
                                             '16512,\n'
                                             '            "80.000000" : '
                                             '18560,\n'
                                             '            "90.000000" : '
                                             '20096,\n'
                                             '            "95.000000" : '
                                             '22144,\n'
                                             '            "99.000000" : '
                                             '31616,\n'
                                             '            "99.500000" : '
                                             '35584,\n'
                                             '            "99.900000" : '
                                             '48384,\n'
                                             '            "99.950000" : '
                                             '62720,\n'
                                             '            "99.990000" : 78336\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5080,\n'
                                             '          "max" : 130805,\n'
                                             '          "mean" : '
                                             '14613.700084,\n'
                                             '          "stddev" : '
                                             '5209.006901,\n'
                                             '          "N" : 3604630\n'
                                             '        },\n'
                                             '        "bw_min" : 256603,\n'
                                             '        "bw_max" : 259025,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '257567.500000,\n'
                                             '        "bw_dev" : 671.520634,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 513206,\n'
                                             '        "iops_max" : 518050,\n'
                                             '        "iops_mean" : '
                                             '515135.142857,\n'
                                             '        "iops_stddev" : '
                                             '1342.858940,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 26,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 9.715921,\n'
                                             '        "4" : 43.387421,\n'
                                             '        "8" : 46.894910,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 95.677259,\n'
                                             '        "8" : 4.322741,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 22.285450,\n'
                                             '        "20" : 67.773863,\n'
                                             '        "50" : 9.846531,\n'
                                             '        "100" : 0.092104,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=143310: Wed Oct 26 04:47:36 '
                                             '2022\n'
                                             '  read: IOPS=515k, BW=251MiB/s '
                                             '(264MB/s)(1760MiB/7001msec)\n'
                                             '    slat (nsec): min=170, '
                                             'max=9297, avg=249.73, '
                                             'stdev=28.58\n'
                                             '    clat (usec): min=4, max=130, '
                                             'avg=14.36, stdev= 5.22\n'
                                             '     lat (usec): min=5, max=130, '
                                             'avg=14.61, stdev= 5.21\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 7456],  '
                                             '5.00th=[ 8384], 10.00th=[ 8896], '
                                             '20.00th=[ 9792],\n'
                                             '     | 30.00th=[10560], '
                                             '40.00th=[11712], '
                                             '50.00th=[14144], '
                                             '60.00th=[15424],\n'
                                             '     | 70.00th=[16512], '
                                             '80.00th=[18560], '
                                             '90.00th=[20096], '
                                             '95.00th=[22144],\n'
                                             '     | 99.00th=[31616], '
                                             '99.50th=[35584], '
                                             '99.90th=[48384], '
                                             '99.95th=[62720],\n'
                                             '     | 99.99th=[78336]\n'
                                             '   bw (  KiB/s): min=256603, '
                                             'max=259025, per=100.00%, '
                                             'avg=257567.50, stdev=671.52, '
                                             'samples=14\n'
                                             '   iops        : min=513206, '
                                             'max=518050, avg=515135.14, '
                                             'stdev=1342.86, samples=14\n'
                                             '  lat (usec)   : 10=22.29%, '
                                             '20=67.77%, 50=9.85%, 100=0.09%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=26, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=9.7%, '
                                             '4=43.4%, 8=46.9%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=95.7%, 8=4.3%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=3604630,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=251MiB/s (264MB/s), '
                                             '251MiB/s-251MiB/s '
                                             '(264MB/s-264MB/s), io=1760MiB '
                                             '(1846MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.output'),
                                   'state': {'begin': 1666745367.241164,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.819014310836792,
                                             'end': 1666745368.0601783,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_321.state')},
                       'cmd_322': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.output'),
                                   'state': {'begin': 1666745370.0823493,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7883975505828857,
                                             'end': 1666745370.8707469,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_322.state')},
                       'cmd_323': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.output'),
                                   'state': {'begin': 1666745370.872893,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.4997878074646,
                                             'end': 1666745382.372681,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_323.state')},
                       'cmd_324': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752471,\n'
                                             '  "timestamp_ms" : '
                                             '1666752471482,\n'
                                             '  "time" : "Wed Oct 26 04:47:51 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2261216256,\n'
                                             '        "io_kbytes" : 2208219,\n'
                                             '        "bw_bytes" : 322984753,\n'
                                             '        "bw" : 315414,\n'
                                             '        "iops" : 630828.595915,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 4416431,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 190,\n'
                                             '          "max" : 24937,\n'
                                             '          "mean" : 265.485439,\n'
                                             '          "stddev" : 66.086546,\n'
                                             '          "N" : 4416431\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 921,\n'
                                             '          "max" : 131396,\n'
                                             '          "mean" : '
                                             '12004.528837,\n'
                                             '          "stddev" : '
                                             '4455.386441,\n'
                                             '          "N" : 4416438,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 6816,\n'
                                             '            "5.000000" : 7904,\n'
                                             '            "10.000000" : 8512,\n'
                                             '            "20.000000" : 9536,\n'
                                             '            "30.000000" : '
                                             '10176,\n'
                                             '            "40.000000" : '
                                             '10688,\n'
                                             '            "50.000000" : '
                                             '11072,\n'
                                             '            "60.000000" : '
                                             '11712,\n'
                                             '            "70.000000" : '
                                             '12480,\n'
                                             '            "80.000000" : '
                                             '13248,\n'
                                             '            "90.000000" : '
                                             '14784,\n'
                                             '            "95.000000" : '
                                             '19840,\n'
                                             '            "99.000000" : '
                                             '31104,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '49920,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5580,\n'
                                             '          "max" : 131617,\n'
                                             '          "mean" : '
                                             '12270.014258,\n'
                                             '          "stddev" : '
                                             '4453.681752,\n'
                                             '          "N" : 4416438\n'
                                             '        },\n'
                                             '        "bw_min" : 314606,\n'
                                             '        "bw_max" : 317057,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '315565.928571,\n'
                                             '        "bw_dev" : 753.328967,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 629212,\n'
                                             '        "iops_max" : 634114,\n'
                                             '        "iops_mean" : '
                                             '631131.928571,\n'
                                             '        "iops_stddev" : '
                                             '1506.587610,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 23,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 25.115280,\n'
                                             '        "8" : 74.875663,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.991752,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 28.039406,\n'
                                             '        "20" : 66.984359,\n'
                                             '        "50" : 4.876879,\n'
                                             '        "100" : 0.097998,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=143360: Wed Oct 26 04:47:51 '
                                             '2022\n'
                                             '  read: IOPS=631k, BW=308MiB/s '
                                             '(323MB/s)(2156MiB/7001msec)\n'
                                             '    slat (nsec): min=190, '
                                             'max=24937, avg=265.49, '
                                             'stdev=66.09\n'
                                             '    clat (nsec): min=921, '
                                             'max=131396, avg=12004.53, '
                                             'stdev=4455.39\n'
                                             '     lat (usec): min=5, max=131, '
                                             'avg=12.27, stdev= 4.45\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 6816],  '
                                             '5.00th=[ 7904], 10.00th=[ 8512], '
                                             '20.00th=[ 9536],\n'
                                             '     | 30.00th=[10176], '
                                             '40.00th=[10688], '
                                             '50.00th=[11072], '
                                             '60.00th=[11712],\n'
                                             '     | 70.00th=[12480], '
                                             '80.00th=[13248], '
                                             '90.00th=[14784], '
                                             '95.00th=[19840],\n'
                                             '     | 99.00th=[31104], '
                                             '99.50th=[34560], '
                                             '99.90th=[49920], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=314606, '
                                             'max=317057, per=100.00%, '
                                             'avg=315565.93, stdev=753.33, '
                                             'samples=14\n'
                                             '   iops        : min=629212, '
                                             'max=634114, avg=631131.93, '
                                             'stdev=1506.59, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=28.04%, 20=66.98%, '
                                             '50=4.88%\n'
                                             '  lat (usec)   : 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=23, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=25.1%, 8=74.9%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=4416431,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=308MiB/s (323MB/s), '
                                             '308MiB/s-308MiB/s '
                                             '(323MB/s-323MB/s), io=2156MiB '
                                             '(2261MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.output'),
                                   'state': {'begin': 1666745382.375356,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.828303337097168,
                                             'end': 1666745383.2036593,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_324.state')},
                       'cmd_325': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.output'),
                                   'state': {'begin': 1666745384.217071,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.779996395111084,
                                             'end': 1666745384.9970675,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_325.state')},
                       'cmd_326': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.output'),
                                   'state': {'begin': 1666745384.99917,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.040830373764038,
                                             'end': 1666745396.0400004,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_326.state')},
                       'cmd_327': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752485,\n'
                                             '  "timestamp_ms" : '
                                             '1666752485218,\n'
                                             '  "time" : "Wed Oct 26 04:48:05 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1124852224,\n'
                                             '        "io_kbytes" : 1098488,\n'
                                             '        "bw_bytes" : 160670221,\n'
                                             '        "bw" : 156904,\n'
                                             '        "iops" : 313807.884588,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2196969,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1282,\n'
                                             '          "max" : 42510,\n'
                                             '          "mean" : 2472.162358,\n'
                                             '          "stddev" : '
                                             '910.605618,\n'
                                             '          "N" : 2196970\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 80,\n'
                                             '          "max" : 152536,\n'
                                             '          "mean" : '
                                             '22632.731950,\n'
                                             '          "stddev" : '
                                             '3858.580981,\n'
                                             '          "N" : 2196976,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 15552,\n'
                                             '            "5.000000" : 20352,\n'
                                             '            "10.000000" : '
                                             '20352,\n'
                                             '            "20.000000" : '
                                             '20864,\n'
                                             '            "30.000000" : '
                                             '21632,\n'
                                             '            "40.000000" : '
                                             '21888,\n'
                                             '            "50.000000" : '
                                             '22144,\n'
                                             '            "60.000000" : '
                                             '22400,\n'
                                             '            "70.000000" : '
                                             '23168,\n'
                                             '            "80.000000" : '
                                             '23424,\n'
                                             '            "90.000000" : '
                                             '23936,\n'
                                             '            "95.000000" : '
                                             '26496,\n'
                                             '            "99.000000" : '
                                             '42752,\n'
                                             '            "99.500000" : '
                                             '45312,\n'
                                             '            "99.900000" : '
                                             '54016,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 81408\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6793,\n'
                                             '          "max" : 153869,\n'
                                             '          "mean" : '
                                             '25104.895035,\n'
                                             '          "stddev" : '
                                             '3768.630555,\n'
                                             '          "N" : 2196976\n'
                                             '        },\n'
                                             '        "bw_min" : 156481,\n'
                                             '        "bw_max" : 157772,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '156978.923077,\n'
                                             '        "bw_dev" : 443.567819,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 312962,\n'
                                             '        "iops_max" : 315544,\n'
                                             '        "iops_mean" : '
                                             '313957.923077,\n'
                                             '        "iops_stddev" : '
                                             '887.118036,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.042857,\n'
                                             '      "sys_cpu" : 82.785714,\n'
                                             '      "ctx" : 12432,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999954,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.022486,\n'
                                             '        "20" : 3.535780,\n'
                                             '        "50" : 96.305455,\n'
                                             '        "100" : 0.129815,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 3157749,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 17441,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 17441,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=143399: Wed '
                                             'Oct 26 04:48:05 2022\n'
                                             '  read: IOPS=314k, BW=153MiB/s '
                                             '(161MB/s)(1073MiB/7001msec)\n'
                                             '    slat (nsec): min=1282, '
                                             'max=42510, avg=2472.16, '
                                             'stdev=910.61\n'
                                             '    clat (nsec): min=80, '
                                             'max=152536, avg=22632.73, '
                                             'stdev=3858.58\n'
                                             '     lat (usec): min=6, max=153, '
                                             'avg=25.10, stdev= 3.77\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[15552],  '
                                             '5.00th=[20352], 10.00th=[20352], '
                                             '20.00th=[20864],\n'
                                             '     | 30.00th=[21632], '
                                             '40.00th=[21888], '
                                             '50.00th=[22144], '
                                             '60.00th=[22400],\n'
                                             '     | 70.00th=[23168], '
                                             '80.00th=[23424], '
                                             '90.00th=[23936], '
                                             '95.00th=[26496],\n'
                                             '     | 99.00th=[42752], '
                                             '99.50th=[45312], '
                                             '99.90th=[54016], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[81408]\n'
                                             '   bw (  KiB/s): min=156481, '
                                             'max=157772, per=100.00%, '
                                             'avg=156978.92, stdev=443.57, '
                                             'samples=13\n'
                                             '   iops        : min=312962, '
                                             'max=315544, avg=313957.92, '
                                             'stdev=887.12, samples=13\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.02%, 20=3.54%, '
                                             '50=96.31%\n'
                                             '  lat (usec)   : 100=0.13%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=15.04%, '
                                             'sys=82.79%, ctx=12432, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2196969,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=153MiB/s (161MB/s), '
                                             '153MiB/s-153MiB/s '
                                             '(161MB/s-161MB/s), io=1073MiB '
                                             '(1125MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=3157749/0, '
                                             'merge=0/0, ticks=17441/0, '
                                             'in_queue=17441, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.output'),
                                   'state': {'begin': 1666745396.042881,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8225052356719971,
                                             'end': 1666745396.8653862,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_327.state')},
                       'cmd_328': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.output'),
                                   'state': {'begin': 1666745397.8776748,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7909982204437256,
                                             'end': 1666745398.668673,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_328.state')},
                       'cmd_329': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.output'),
                                   'state': {'begin': 1666745398.6709023,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.03939437866211,
                                             'end': 1666745409.7102966,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_329.state')},
                       'cmd_33': {'output': 'io_uring-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751080,\n'
                                            '  "timestamp_ms" : '
                                            '1666751080821,\n'
                                            '  "time" : "Wed Oct 26 04:24:40 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "io_uring",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 284537856,\n'
                                            '        "io_kbytes" : 277869,\n'
                                            '        "bw_bytes" : 40642459,\n'
                                            '        "bw" : 39689,\n'
                                            '        "iops" : 79379.802885,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 555738,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2424,\n'
                                            '          "max" : 32170,\n'
                                            '          "mean" : 2650.417359,\n'
                                            '          "stddev" : 227.123682,\n'
                                            '          "N" : 555738\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 161,\n'
                                            '          "max" : 84729,\n'
                                            '          "mean" : 9278.076117,\n'
                                            '          "stddev" : '
                                            '1249.290259,\n'
                                            '          "N" : 555738,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8896,\n'
                                            '            "5.000000" : 8896,\n'
                                            '            "10.000000" : 9024,\n'
                                            '            "20.000000" : 9024,\n'
                                            '            "30.000000" : 9024,\n'
                                            '            "40.000000" : 9024,\n'
                                            '            "50.000000" : 9152,\n'
                                            '            "60.000000" : 9152,\n'
                                            '            "70.000000" : 9152,\n'
                                            '            "80.000000" : 9152,\n'
                                            '            "90.000000" : 9280,\n'
                                            '            "95.000000" : 9536,\n'
                                            '            "99.000000" : 15296,\n'
                                            '            "99.500000" : 15936,\n'
                                            '            "99.900000" : 19328,\n'
                                            '            "99.950000" : 27264,\n'
                                            '            "99.990000" : 55552\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 8526,\n'
                                            '          "max" : 87364,\n'
                                            '          "mean" : 11928.493475,\n'
                                            '          "stddev" : '
                                            '1269.367601,\n'
                                            '          "N" : 555738\n'
                                            '        },\n'
                                            '        "bw_min" : 39567,\n'
                                            '        "bw_max" : 39849,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '39706.571429,\n'
                                            '        "bw_dev" : 93.082027,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 79134,\n'
                                            '        "iops_max" : 79698,\n'
                                            '        "iops_mean" : '
                                            '79413.142857,\n'
                                            '        "iops_stddev" : '
                                            '186.164054,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 11.571429,\n'
                                            '      "sys_cpu" : 49.257143,\n'
                                            '      "ctx" : 555625,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 96.871188,\n'
                                            '        "20" : 3.046580,\n'
                                            '        "50" : 0.057581,\n'
                                            '        "100" : 0.016555,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 797598,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 4985,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 4985,\n'
                                            '      "util" : 99.088748\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=139097: Wed '
                                            'Oct 26 04:24:40 2022\n'
                                            '  read: IOPS=79.4k, BW=38.8MiB/s '
                                            '(40.6MB/s)(271MiB/7001msec)\n'
                                            '    slat (nsec): min=2424, '
                                            'max=32170, avg=2650.42, '
                                            'stdev=227.12\n'
                                            '    clat (nsec): min=161, '
                                            'max=84729, avg=9278.08, '
                                            'stdev=1249.29\n'
                                            '     lat (nsec): min=8526, '
                                            'max=87364, avg=11928.49, '
                                            'stdev=1269.37\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8896],  5.00th=[ '
                                            '8896], 10.00th=[ 9024], 20.00th=[ '
                                            '9024],\n'
                                            '     | 30.00th=[ 9024], 40.00th=[ '
                                            '9024], 50.00th=[ 9152], 60.00th=[ '
                                            '9152],\n'
                                            '     | 70.00th=[ 9152], 80.00th=[ '
                                            '9152], 90.00th=[ 9280], 95.00th=[ '
                                            '9536],\n'
                                            '     | 99.00th=[15296], '
                                            '99.50th=[15936], 99.90th=[19328], '
                                            '99.95th=[27264],\n'
                                            '     | 99.99th=[55552]\n'
                                            '   bw (  KiB/s): min=39567, '
                                            'max=39849, per=100.00%, '
                                            'avg=39706.57, stdev=93.08, '
                                            'samples=14\n'
                                            '   iops        : min=79134, '
                                            'max=79698, avg=79413.14, '
                                            'stdev=186.16, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 10=96.87%, '
                                            '20=3.05%, 50=0.06%, 100=0.02%\n'
                                            '  cpu          : usr=11.57%, '
                                            'sys=49.26%, ctx=555625, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=555738,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=38.8MiB/s (40.6MB/s), '
                                            '38.8MiB/s-38.8MiB/s '
                                            '(40.6MB/s-40.6MB/s), io=271MiB '
                                            '(285MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=797598/0, '
                                            'merge=0/0, ticks=4985/0, '
                                            'in_queue=4985, util=99.09%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.output'),
                                  'state': {'begin': 1666743991.6221285,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8044507503509521,
                                            'end': 1666743992.4265792,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_33.state')},
                       'cmd_330': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752498,\n'
                                             '  "timestamp_ms" : '
                                             '1666752498888,\n'
                                             '  "time" : "Wed Oct 26 04:48:18 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1118396416,\n'
                                             '        "io_kbytes" : 1092184,\n'
                                             '        "bw_bytes" : 159748095,\n'
                                             '        "bw" : 156003,\n'
                                             '        "iops" : 312006.999000,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2184361,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1282,\n'
                                             '          "max" : 29055,\n'
                                             '          "mean" : 2484.140099,\n'
                                             '          "stddev" : '
                                             '984.021916,\n'
                                             '          "N" : 2184361\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 70,\n'
                                             '          "max" : 126137,\n'
                                             '          "mean" : '
                                             '22762.563038,\n'
                                             '          "stddev" : '
                                             '3843.773518,\n'
                                             '          "N" : 2184368,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 15936,\n'
                                             '            "5.000000" : 20352,\n'
                                             '            "10.000000" : '
                                             '20608,\n'
                                             '            "20.000000" : '
                                             '20864,\n'
                                             '            "30.000000" : '
                                             '21632,\n'
                                             '            "40.000000" : '
                                             '21888,\n'
                                             '            "50.000000" : '
                                             '22144,\n'
                                             '            "60.000000" : '
                                             '22656,\n'
                                             '            "70.000000" : '
                                             '23424,\n'
                                             '            "80.000000" : '
                                             '23680,\n'
                                             '            "90.000000" : '
                                             '23936,\n'
                                             '            "95.000000" : '
                                             '26752,\n'
                                             '            "99.000000" : '
                                             '42752,\n'
                                             '            "99.500000" : '
                                             '45312,\n'
                                             '            "99.900000" : '
                                             '54528,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 80384\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6782,\n'
                                             '          "max" : 127519,\n'
                                             '          "mean" : '
                                             '25246.704148,\n'
                                             '          "stddev" : '
                                             '3767.691495,\n'
                                             '          "N" : 2184368\n'
                                             '        },\n'
                                             '        "bw_min" : 155626,\n'
                                             '        "bw_max" : 156619,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '156072.692308,\n'
                                             '        "bw_dev" : 295.570461,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 311252,\n'
                                             '        "iops_max" : 313238,\n'
                                             '        "iops_mean" : '
                                             '312145.692308,\n'
                                             '        "iops_stddev" : '
                                             '591.374160,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.014286,\n'
                                             '      "sys_cpu" : 83.885714,\n'
                                             '      "ctx" : 12410,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999954,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.028109,\n'
                                             '        "20" : 3.435879,\n'
                                             '        "50" : 96.395696,\n'
                                             '        "100" : 0.135875,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 3135878,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 17469,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 17469,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=143437: Wed '
                                             'Oct 26 04:48:18 2022\n'
                                             '  read: IOPS=312k, BW=152MiB/s '
                                             '(160MB/s)(1067MiB/7001msec)\n'
                                             '    slat (nsec): min=1282, '
                                             'max=29055, avg=2484.14, '
                                             'stdev=984.02\n'
                                             '    clat (nsec): min=70, '
                                             'max=126137, avg=22762.56, '
                                             'stdev=3843.77\n'
                                             '     lat (usec): min=6, max=127, '
                                             'avg=25.25, stdev= 3.77\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[15936],  '
                                             '5.00th=[20352], 10.00th=[20608], '
                                             '20.00th=[20864],\n'
                                             '     | 30.00th=[21632], '
                                             '40.00th=[21888], '
                                             '50.00th=[22144], '
                                             '60.00th=[22656],\n'
                                             '     | 70.00th=[23424], '
                                             '80.00th=[23680], '
                                             '90.00th=[23936], '
                                             '95.00th=[26752],\n'
                                             '     | 99.00th=[42752], '
                                             '99.50th=[45312], '
                                             '99.90th=[54528], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[80384]\n'
                                             '   bw (  KiB/s): min=155626, '
                                             'max=156619, per=100.00%, '
                                             'avg=156072.69, stdev=295.57, '
                                             'samples=13\n'
                                             '   iops        : min=311252, '
                                             'max=313238, avg=312145.69, '
                                             'stdev=591.37, samples=13\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.03%, 20=3.44%, '
                                             '50=96.40%\n'
                                             '  lat (usec)   : 100=0.14%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=14.01%, '
                                             'sys=83.89%, ctx=12410, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2184361,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=152MiB/s (160MB/s), '
                                             '152MiB/s-152MiB/s '
                                             '(160MB/s-160MB/s), io=1067MiB '
                                             '(1118MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=3135878/0, '
                                             'merge=0/0, ticks=17469/0, '
                                             'in_queue=17469, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.output'),
                                   'state': {'begin': 1666745409.7132373,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7861456871032715,
                                             'end': 1666745410.499383,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_330.state')},
                       'cmd_331': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.output'),
                                   'state': {'begin': 1666745411.4932039,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7851612567901611,
                                             'end': 1666745412.2783651,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_331.state')},
                       'cmd_332': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.output'),
                                   'state': {'begin': 1666745412.280575,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.046215057373047,
                                             'end': 1666745423.32679,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_332.state')},
                       'cmd_333': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=io_uring, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752512,\n'
                                             '  "timestamp_ms" : '
                                             '1666752512505,\n'
                                             '  "time" : "Wed Oct 26 04:48:32 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1115583488,\n'
                                             '        "io_kbytes" : 1089437,\n'
                                             '        "bw_bytes" : 159346305,\n'
                                             '        "bw" : 155611,\n'
                                             '        "iops" : 311222.111127,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2178866,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1292,\n'
                                             '          "max" : 72896,\n'
                                             '          "mean" : 2499.882950,\n'
                                             '          "stddev" : '
                                             '1010.452035,\n'
                                             '          "N" : 2178867\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 80,\n'
                                             '          "max" : 140895,\n'
                                             '          "mean" : '
                                             '22828.623283,\n'
                                             '          "stddev" : '
                                             '3883.009481,\n'
                                             '          "N" : 2178873,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 15936,\n'
                                             '            "5.000000" : 20352,\n'
                                             '            "10.000000" : '
                                             '20608,\n'
                                             '            "20.000000" : '
                                             '20864,\n'
                                             '            "30.000000" : '
                                             '21632,\n'
                                             '            "40.000000" : '
                                             '21888,\n'
                                             '            "50.000000" : '
                                             '22400,\n'
                                             '            "60.000000" : '
                                             '23168,\n'
                                             '            "70.000000" : '
                                             '23424,\n'
                                             '            "80.000000" : '
                                             '23680,\n'
                                             '            "90.000000" : '
                                             '23936,\n'
                                             '            "95.000000" : '
                                             '26752,\n'
                                             '            "99.000000" : '
                                             '42752,\n'
                                             '            "99.500000" : '
                                             '45312,\n'
                                             '            "99.900000" : '
                                             '57600,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 81408\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6842,\n'
                                             '          "max" : 142227,\n'
                                             '          "mean" : '
                                             '25328.506996,\n'
                                             '          "stddev" : '
                                             '3810.015386,\n'
                                             '          "N" : 2178873\n'
                                             '        },\n'
                                             '        "bw_min" : 155507,\n'
                                             '        "bw_max" : 155865,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '155698.769231,\n'
                                             '        "bw_dev" : 108.354475,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 311014,\n'
                                             '        "iops_max" : 311730,\n'
                                             '        "iops_mean" : '
                                             '311397.538462,\n'
                                             '        "iops_stddev" : '
                                             '216.708951,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.385714,\n'
                                             '      "sys_cpu" : 80.500000,\n'
                                             '      "ctx" : 12497,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999954,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.027262,\n'
                                             '        "20" : 3.408792,\n'
                                             '        "50" : 96.409784,\n'
                                             '        "100" : 0.148013,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 3131246,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 17492,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 17492,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=143476: Wed '
                                             'Oct 26 04:48:32 2022\n'
                                             '  read: IOPS=311k, BW=152MiB/s '
                                             '(159MB/s)(1064MiB/7001msec)\n'
                                             '    slat (nsec): min=1292, '
                                             'max=72896, avg=2499.88, '
                                             'stdev=1010.45\n'
                                             '    clat (nsec): min=80, '
                                             'max=140895, avg=22828.62, '
                                             'stdev=3883.01\n'
                                             '     lat (usec): min=6, max=142, '
                                             'avg=25.33, stdev= 3.81\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[15936],  '
                                             '5.00th=[20352], 10.00th=[20608], '
                                             '20.00th=[20864],\n'
                                             '     | 30.00th=[21632], '
                                             '40.00th=[21888], '
                                             '50.00th=[22400], '
                                             '60.00th=[23168],\n'
                                             '     | 70.00th=[23424], '
                                             '80.00th=[23680], '
                                             '90.00th=[23936], '
                                             '95.00th=[26752],\n'
                                             '     | 99.00th=[42752], '
                                             '99.50th=[45312], '
                                             '99.90th=[57600], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[81408]\n'
                                             '   bw (  KiB/s): min=155507, '
                                             'max=155865, per=100.00%, '
                                             'avg=155698.77, stdev=108.35, '
                                             'samples=13\n'
                                             '   iops        : min=311014, '
                                             'max=311730, avg=311397.54, '
                                             'stdev=216.71, samples=13\n'
                                             '  lat (nsec)   : 100=0.01%, '
                                             '250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.03%, 20=3.41%, '
                                             '50=96.41%\n'
                                             '  lat (usec)   : 100=0.15%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=17.39%, '
                                             'sys=80.50%, ctx=12497, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2178866,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=152MiB/s (159MB/s), '
                                             '152MiB/s-152MiB/s '
                                             '(159MB/s-159MB/s), io=1064MiB '
                                             '(1116MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=3131246/0, '
                                             'merge=0/0, ticks=17492/0, '
                                             'in_queue=17492, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.output'),
                                   'state': {'begin': 1666745423.3296907,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.78910231590271,
                                             'end': 1666745424.118793,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_333.state')},
                       'cmd_334': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.output'),
                                   'state': {'begin': 1666745426.137515,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.786327600479126,
                                             'end': 1666745426.9238427,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_334.state')},
                       'cmd_335': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.output'),
                                   'state': {'begin': 1666745426.9262161,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.461182594299316,
                                             'end': 1666745438.3873987,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_335.state')},
                       'cmd_336': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752527,\n'
                                             '  "timestamp_ms" : '
                                             '1666752527493,\n'
                                             '  "time" : "Wed Oct 26 04:48:47 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1236315648,\n'
                                             '        "io_kbytes" : 1207339,\n'
                                             '        "bw_bytes" : 176591293,\n'
                                             '        "bw" : 172452,\n'
                                             '        "iops" : 344903.870876,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2414672,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 120,\n'
                                             '          "max" : 12142,\n'
                                             '          "mean" : 141.289363,\n'
                                             '          "stddev" : 45.104120,\n'
                                             '          "N" : 2414672\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 19116,\n'
                                             '          "max" : 118853,\n'
                                             '          "mean" : '
                                             '21467.040025,\n'
                                             '          "stddev" : '
                                             '2785.766655,\n'
                                             '          "N" : 2414672,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 19584,\n'
                                             '            "5.000000" : 19840,\n'
                                             '            "10.000000" : '
                                             '20096,\n'
                                             '            "20.000000" : '
                                             '20096,\n'
                                             '            "30.000000" : '
                                             '20352,\n'
                                             '            "40.000000" : '
                                             '20608,\n'
                                             '            "50.000000" : '
                                             '20864,\n'
                                             '            "60.000000" : '
                                             '21120,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21632,\n'
                                             '            "90.000000" : '
                                             '21888,\n'
                                             '            "95.000000" : '
                                             '27520,\n'
                                             '            "99.000000" : '
                                             '33536,\n'
                                             '            "99.500000" : '
                                             '37632,\n'
                                             '            "99.900000" : '
                                             '49920,\n'
                                             '            "99.950000" : '
                                             '60672,\n'
                                             '            "99.990000" : 71168\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 19247,\n'
                                             '          "max" : 119083,\n'
                                             '          "mean" : '
                                             '21608.329388,\n'
                                             '          "stddev" : '
                                             '2789.818426,\n'
                                             '          "N" : 2414672\n'
                                             '        },\n'
                                             '        "bw_min" : 172136,\n'
                                             '        "bw_max" : 172888,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '172497.071429,\n'
                                             '        "bw_dev" : 232.881767,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 344272,\n'
                                             '        "iops_max" : 345776,\n'
                                             '        "iops_mean" : '
                                             '344994.142857,\n'
                                             '        "iops_stddev" : '
                                             '465.763535,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 43.014286,\n'
                                             '      "sys_cpu" : 56.957143,\n'
                                             '      "ctx" : 21,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 12.500000,\n'
                                             '        "2" : 25.000000,\n'
                                             '        "4" : 50.000000,\n'
                                             '        "8" : 12.500000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 88.888889,\n'
                                             '        "8" : 11.111111,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 11.204420,\n'
                                             '        "50" : 88.695897,\n'
                                             '        "100" : 0.098978,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=143526: Wed '
                                             'Oct 26 04:48:47 2022\n'
                                             '  read: IOPS=345k, BW=168MiB/s '
                                             '(177MB/s)(1179MiB/7001msec)\n'
                                             '    slat (nsec): min=120, '
                                             'max=12142, avg=141.29, '
                                             'stdev=45.10\n'
                                             '    clat (usec): min=19, '
                                             'max=118, avg=21.47, stdev= 2.79\n'
                                             '     lat (usec): min=19, '
                                             'max=119, avg=21.61, stdev= 2.79\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[19584],  '
                                             '5.00th=[19840], 10.00th=[20096], '
                                             '20.00th=[20096],\n'
                                             '     | 30.00th=[20352], '
                                             '40.00th=[20608], '
                                             '50.00th=[20864], '
                                             '60.00th=[21120],\n'
                                             '     | 70.00th=[21376], '
                                             '80.00th=[21632], '
                                             '90.00th=[21888], '
                                             '95.00th=[27520],\n'
                                             '     | 99.00th=[33536], '
                                             '99.50th=[37632], '
                                             '99.90th=[49920], '
                                             '99.95th=[60672],\n'
                                             '     | 99.99th=[71168]\n'
                                             '   bw (  KiB/s): min=172136, '
                                             'max=172888, per=100.00%, '
                                             'avg=172497.07, stdev=232.88, '
                                             'samples=14\n'
                                             '   iops        : min=344272, '
                                             'max=345776, avg=344994.14, '
                                             'stdev=465.76, samples=14\n'
                                             '  lat (usec)   : 20=11.20%, '
                                             '50=88.70%, 100=0.10%, 250=0.01%\n'
                                             '  cpu          : usr=43.01%, '
                                             'sys=56.96%, ctx=21, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=12.5%, '
                                             '2=25.0%, 4=50.0%, 8=12.5%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=88.9%, 8=11.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2414672,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=168MiB/s (177MB/s), '
                                             '168MiB/s-168MiB/s '
                                             '(177MB/s-177MB/s), io=1179MiB '
                                             '(1236MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.output'),
                                   'state': {'begin': 1666745438.390226,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.804879903793335,
                                             'end': 1666745439.1951058,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_336.state')},
                       'cmd_337': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.output'),
                                   'state': {'begin': 1666745441.2478127,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7863655090332031,
                                             'end': 1666745442.0341783,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_337.state')},
                       'cmd_338': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.output'),
                                   'state': {'begin': 1666745442.0363488,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.445159196853638,
                                             'end': 1666745453.481508,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_338.state')},
                       'cmd_339': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752542,\n'
                                             '  "timestamp_ms" : '
                                             '1666752542587,\n'
                                             '  "time" : "Wed Oct 26 04:49:02 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1230662144,\n'
                                             '        "io_kbytes" : 1201818,\n'
                                             '        "bw_bytes" : 175783765,\n'
                                             '        "bw" : 171663,\n'
                                             '        "iops" : 343326.667619,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2403630,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 24797,\n'
                                             '          "mean" : 132.649914,\n'
                                             '          "stddev" : 44.296597,\n'
                                             '          "N" : 2403630\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 16370,\n'
                                             '          "max" : 105507,\n'
                                             '          "mean" : '
                                             '21585.856196,\n'
                                             '          "stddev" : '
                                             '2810.338268,\n'
                                             '          "N" : 2403630,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 19584,\n'
                                             '            "5.000000" : 19840,\n'
                                             '            "10.000000" : '
                                             '20096,\n'
                                             '            "20.000000" : '
                                             '20352,\n'
                                             '            "30.000000" : '
                                             '20608,\n'
                                             '            "40.000000" : '
                                             '20864,\n'
                                             '            "50.000000" : '
                                             '21120,\n'
                                             '            "60.000000" : '
                                             '21120,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21632,\n'
                                             '            "90.000000" : '
                                             '22144,\n'
                                             '            "95.000000" : '
                                             '27008,\n'
                                             '            "99.000000" : '
                                             '35584,\n'
                                             '            "99.500000" : '
                                             '38144,\n'
                                             '            "99.900000" : '
                                             '51456,\n'
                                             '            "99.950000" : '
                                             '61184,\n'
                                             '            "99.990000" : 71168\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 16490,\n'
                                             '          "max" : 105708,\n'
                                             '          "mean" : '
                                             '21718.506110,\n'
                                             '          "stddev" : '
                                             '2814.226828,\n'
                                             '          "N" : 2403630\n'
                                             '        },\n'
                                             '        "bw_min" : 171328,\n'
                                             '        "bw_max" : 172104,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '171710.785714,\n'
                                             '        "bw_dev" : 208.705231,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 342656,\n'
                                             '        "iops_max" : 344208,\n'
                                             '        "iops_mean" : '
                                             '343421.571429,\n'
                                             '        "iops_stddev" : '
                                             '417.410463,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.128571,\n'
                                             '      "sys_cpu" : 60.842857,\n'
                                             '      "ctx" : 23,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 12.500010,\n'
                                             '        "2" : 25.000021,\n'
                                             '        "4" : 50.000000,\n'
                                             '        "8" : 12.499969,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 88.888881,\n'
                                             '        "8" : 11.111119,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 7.242213,\n'
                                             '        "50" : 92.644500,\n'
                                             '        "100" : 0.112455,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=143575: Wed '
                                             'Oct 26 04:49:02 2022\n'
                                             '  read: IOPS=343k, BW=168MiB/s '
                                             '(176MB/s)(1174MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=24797, avg=132.65, '
                                             'stdev=44.30\n'
                                             '    clat (usec): min=16, '
                                             'max=105, avg=21.59, stdev= 2.81\n'
                                             '     lat (usec): min=16, '
                                             'max=105, avg=21.72, stdev= 2.81\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[19584],  '
                                             '5.00th=[19840], 10.00th=[20096], '
                                             '20.00th=[20352],\n'
                                             '     | 30.00th=[20608], '
                                             '40.00th=[20864], '
                                             '50.00th=[21120], '
                                             '60.00th=[21120],\n'
                                             '     | 70.00th=[21376], '
                                             '80.00th=[21632], '
                                             '90.00th=[22144], '
                                             '95.00th=[27008],\n'
                                             '     | 99.00th=[35584], '
                                             '99.50th=[38144], '
                                             '99.90th=[51456], '
                                             '99.95th=[61184],\n'
                                             '     | 99.99th=[71168]\n'
                                             '   bw (  KiB/s): min=171328, '
                                             'max=172104, per=100.00%, '
                                             'avg=171710.79, stdev=208.71, '
                                             'samples=14\n'
                                             '   iops        : min=342656, '
                                             'max=344208, avg=343421.57, '
                                             'stdev=417.41, samples=14\n'
                                             '  lat (usec)   : 20=7.24%, '
                                             '50=92.64%, 100=0.11%, 250=0.01%\n'
                                             '  cpu          : usr=39.13%, '
                                             'sys=60.84%, ctx=23, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=12.5%, '
                                             '2=25.0%, 4=50.0%, 8=12.5%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=88.9%, 8=11.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2403630,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=168MiB/s (176MB/s), '
                                             '168MiB/s-168MiB/s '
                                             '(176MB/s-176MB/s), io=1174MiB '
                                             '(1231MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.output'),
                                   'state': {'begin': 1666745453.484458,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7894268035888672,
                                             'end': 1666745454.2738848,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_339.state')},
                       'cmd_34': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.output'),
                                  'state': {'begin': 1666743993.4221244,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7811994552612305,
                                            'end': 1666743994.2033238,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_34.state')},
                       'cmd_340': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.output'),
                                   'state': {'begin': 1666745456.3527138,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7916009426116943,
                                             'end': 1666745457.1443148,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_340.state')},
                       'cmd_341': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.output'),
                                   'state': {'begin': 1666745457.1465938,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.462625741958618,
                                             'end': 1666745468.6092196,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_341.state')},
                       'cmd_342': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752557,\n'
                                             '  "timestamp_ms" : '
                                             '1666752557715,\n'
                                             '  "time" : "Wed Oct 26 04:49:17 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1238494720,\n'
                                             '        "io_kbytes" : 1209467,\n'
                                             '        "bw_bytes" : 176902545,\n'
                                             '        "bw" : 172756,\n'
                                             '        "iops" : 345511.784031,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2418928,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 11802,\n'
                                             '          "mean" : 126.931602,\n'
                                             '          "stddev" : 38.723132,\n'
                                             '          "N" : 2418928\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 19076,\n'
                                             '          "max" : 118472,\n'
                                             '          "mean" : '
                                             '21508.297008,\n'
                                             '          "stddev" : '
                                             '2755.141185,\n'
                                             '          "N" : 2418928,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 19584,\n'
                                             '            "5.000000" : 19840,\n'
                                             '            "10.000000" : '
                                             '20096,\n'
                                             '            "20.000000" : '
                                             '20352,\n'
                                             '            "30.000000" : '
                                             '20608,\n'
                                             '            "40.000000" : '
                                             '20864,\n'
                                             '            "50.000000" : '
                                             '20864,\n'
                                             '            "60.000000" : '
                                             '21120,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21632,\n'
                                             '            "90.000000" : '
                                             '22144,\n'
                                             '            "95.000000" : '
                                             '27008,\n'
                                             '            "99.000000" : '
                                             '33536,\n'
                                             '            "99.500000" : '
                                             '37632,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '60672,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 19196,\n'
                                             '          "max" : 118672,\n'
                                             '          "mean" : '
                                             '21635.228610,\n'
                                             '          "stddev" : '
                                             '2758.482082,\n'
                                             '          "N" : 2418928\n'
                                             '        },\n'
                                             '        "bw_min" : 172512,\n'
                                             '        "bw_max" : 173032,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '172796.571429,\n'
                                             '        "bw_dev" : 170.218735,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 345024,\n'
                                             '        "iops_max" : 346064,\n'
                                             '        "iops_mean" : '
                                             '345593.214286,\n'
                                             '        "iops_stddev" : '
                                             '340.424662,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 44.100000,\n'
                                             '      "sys_cpu" : 55.871429,\n'
                                             '      "ctx" : 22,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 12.500000,\n'
                                             '        "2" : 25.000000,\n'
                                             '        "4" : 50.000000,\n'
                                             '        "8" : 12.500000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 88.888889,\n'
                                             '        "8" : 11.111111,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 7.970183,\n'
                                             '        "50" : 91.925431,\n'
                                             '        "100" : 0.102401,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=143624: Wed '
                                             'Oct 26 04:49:17 2022\n'
                                             '  read: IOPS=346k, BW=169MiB/s '
                                             '(177MB/s)(1181MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=11802, avg=126.93, '
                                             'stdev=38.72\n'
                                             '    clat (usec): min=19, '
                                             'max=118, avg=21.51, stdev= 2.76\n'
                                             '     lat (usec): min=19, '
                                             'max=118, avg=21.64, stdev= 2.76\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[19584],  '
                                             '5.00th=[19840], 10.00th=[20096], '
                                             '20.00th=[20352],\n'
                                             '     | 30.00th=[20608], '
                                             '40.00th=[20864], '
                                             '50.00th=[20864], '
                                             '60.00th=[21120],\n'
                                             '     | 70.00th=[21376], '
                                             '80.00th=[21632], '
                                             '90.00th=[22144], '
                                             '95.00th=[27008],\n'
                                             '     | 99.00th=[33536], '
                                             '99.50th=[37632], '
                                             '99.90th=[50432], '
                                             '99.95th=[60672],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=172512, '
                                             'max=173032, per=100.00%, '
                                             'avg=172796.57, stdev=170.22, '
                                             'samples=14\n'
                                             '   iops        : min=345024, '
                                             'max=346064, avg=345593.21, '
                                             'stdev=340.42, samples=14\n'
                                             '  lat (usec)   : 20=7.97%, '
                                             '50=91.93%, 100=0.10%, 250=0.01%\n'
                                             '  cpu          : usr=44.10%, '
                                             'sys=55.87%, ctx=22, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=12.5%, '
                                             '2=25.0%, 4=50.0%, 8=12.5%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=88.9%, 8=11.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2418928,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=169MiB/s (177MB/s), '
                                             '169MiB/s-169MiB/s '
                                             '(177MB/s-177MB/s), io=1181MiB '
                                             '(1238MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.output'),
                                   'state': {'begin': 1666745468.6123612,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.795048713684082,
                                             'end': 1666745469.40741,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_342.state')},
                       'cmd_343': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.output'),
                                   'state': {'begin': 1666745471.5778756,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7822365760803223,
                                             'end': 1666745472.3601122,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_343.state')},
                       'cmd_344': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.output'),
                                   'state': {'begin': 1666745472.3624802,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.46537733078003,
                                             'end': 1666745483.8278575,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_344.state')},
                       'cmd_345': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752572,\n'
                                             '  "timestamp_ms" : '
                                             '1666752572934,\n'
                                             '  "time" : "Wed Oct 26 04:49:32 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1087555072,\n'
                                             '        "io_kbytes" : 1062065,\n'
                                             '        "bw_bytes" : 155342818,\n'
                                             '        "bw" : 151701,\n'
                                             '        "iops" : 303402.942437,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2124124,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1522,\n'
                                             '          "max" : 29946,\n'
                                             '          "mean" : 2805.505978,\n'
                                             '          "stddev" : '
                                             '1163.675313,\n'
                                             '          "N" : 2124124\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 190,\n'
                                             '          "max" : 125075,\n'
                                             '          "mean" : '
                                             '23326.719890,\n'
                                             '          "stddev" : '
                                             '2999.627077,\n'
                                             '          "N" : 2124131,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16768,\n'
                                             '            "5.000000" : 21376,\n'
                                             '            "10.000000" : '
                                             '21632,\n'
                                             '            "20.000000" : '
                                             '21888,\n'
                                             '            "30.000000" : '
                                             '22656,\n'
                                             '            "40.000000" : '
                                             '22912,\n'
                                             '            "50.000000" : '
                                             '23168,\n'
                                             '            "60.000000" : '
                                             '23168,\n'
                                             '            "70.000000" : '
                                             '23680,\n'
                                             '            "80.000000" : '
                                             '24192,\n'
                                             '            "90.000000" : '
                                             '24448,\n'
                                             '            "95.000000" : '
                                             '27264,\n'
                                             '            "99.000000" : '
                                             '34560,\n'
                                             '            "99.500000" : '
                                             '40192,\n'
                                             '            "99.900000" : '
                                             '55040,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6311,\n'
                                             '          "max" : 126648,\n'
                                             '          "mean" : '
                                             '26132.225886,\n'
                                             '          "stddev" : '
                                             '2927.082286,\n'
                                             '          "N" : 2124131\n'
                                             '        },\n'
                                             '        "bw_min" : 151465,\n'
                                             '        "bw_max" : 152209,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '151742.642857,\n'
                                             '        "bw_dev" : 193.711366,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 302930,\n'
                                             '        "iops_max" : 304420,\n'
                                             '        "iops_mean" : '
                                             '303485.500000,\n'
                                             '        "iops_stddev" : '
                                             '388.155550,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 31.928571,\n'
                                             '      "sys_cpu" : 68.042857,\n'
                                             '      "ctx" : 26,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999976,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.030177,\n'
                                             '        "20" : 2.877798,\n'
                                             '        "50" : 96.939209,\n'
                                             '        "100" : 0.136903,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=143673: Wed '
                                             'Oct 26 04:49:32 2022\n'
                                             '  read: IOPS=303k, BW=148MiB/s '
                                             '(155MB/s)(1037MiB/7001msec)\n'
                                             '    slat (nsec): min=1522, '
                                             'max=29946, avg=2805.51, '
                                             'stdev=1163.68\n'
                                             '    clat (nsec): min=190, '
                                             'max=125075, avg=23326.72, '
                                             'stdev=2999.63\n'
                                             '     lat (usec): min=6, max=126, '
                                             'avg=26.13, stdev= 2.93\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16768],  '
                                             '5.00th=[21376], 10.00th=[21632], '
                                             '20.00th=[21888],\n'
                                             '     | 30.00th=[22656], '
                                             '40.00th=[22912], '
                                             '50.00th=[23168], '
                                             '60.00th=[23168],\n'
                                             '     | 70.00th=[23680], '
                                             '80.00th=[24192], '
                                             '90.00th=[24448], '
                                             '95.00th=[27264],\n'
                                             '     | 99.00th=[34560], '
                                             '99.50th=[40192], '
                                             '99.90th=[55040], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=151465, '
                                             'max=152209, per=100.00%, '
                                             'avg=151742.64, stdev=193.71, '
                                             'samples=14\n'
                                             '   iops        : min=302930, '
                                             'max=304420, avg=303485.50, '
                                             'stdev=388.16, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.03%, 20=2.88%, 50=96.94%, '
                                             '100=0.14%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=31.93%, '
                                             'sys=68.04%, ctx=26, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2124124,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=148MiB/s (155MB/s), '
                                             '148MiB/s-148MiB/s '
                                             '(155MB/s-155MB/s), io=1037MiB '
                                             '(1088MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.output'),
                                   'state': {'begin': 1666745483.8309026,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7939887046813965,
                                             'end': 1666745484.6248913,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_345.state')},
                       'cmd_346': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.output'),
                                   'state': {'begin': 1666745486.6498613,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7868752479553223,
                                             'end': 1666745487.4367366,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_346.state')},
                       'cmd_347': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.output'),
                                   'state': {'begin': 1666745487.4388833,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.456100225448608,
                                             'end': 1666745498.8949835,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_347.state')},
                       'cmd_348': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752588,\n'
                                             '  "timestamp_ms" : '
                                             '1666752588001,\n'
                                             '  "time" : "Wed Oct 26 04:49:48 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1087149056,\n'
                                             '        "io_kbytes" : 1061669,\n'
                                             '        "bw_bytes" : 155284824,\n'
                                             '        "bw" : 151645,\n'
                                             '        "iops" : 303289.530067,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2123330,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1523,\n'
                                             '          "max" : 35066,\n'
                                             '          "mean" : 2799.298077,\n'
                                             '          "stddev" : '
                                             '1164.539197,\n'
                                             '          "N" : 2123331\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 200,\n'
                                             '          "max" : 131516,\n'
                                             '          "mean" : '
                                             '23335.457212,\n'
                                             '          "stddev" : '
                                             '2973.486735,\n'
                                             '          "N" : 2123337,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16768,\n'
                                             '            "5.000000" : 21376,\n'
                                             '            "10.000000" : '
                                             '21632,\n'
                                             '            "20.000000" : '
                                             '21888,\n'
                                             '            "30.000000" : '
                                             '22656,\n'
                                             '            "40.000000" : '
                                             '22912,\n'
                                             '            "50.000000" : '
                                             '23168,\n'
                                             '            "60.000000" : '
                                             '23168,\n'
                                             '            "70.000000" : '
                                             '23680,\n'
                                             '            "80.000000" : '
                                             '24192,\n'
                                             '            "90.000000" : '
                                             '24448,\n'
                                             '            "95.000000" : '
                                             '27008,\n'
                                             '            "99.000000" : '
                                             '34560,\n'
                                             '            "99.500000" : '
                                             '39680,\n'
                                             '            "99.900000" : '
                                             '54528,\n'
                                             '            "99.950000" : '
                                             '64768,\n'
                                             '            "99.990000" : 76288\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6322,\n'
                                             '          "max" : 135254,\n'
                                             '          "mean" : '
                                             '26134.754886,\n'
                                             '          "stddev" : '
                                             '2903.757258,\n'
                                             '          "N" : 2123337\n'
                                             '        },\n'
                                             '        "bw_min" : 151360,\n'
                                             '        "bw_max" : 152173,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '151693.142857,\n'
                                             '        "bw_dev" : 212.960761,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 302720,\n'
                                             '        "iops_max" : 304346,\n'
                                             '        "iops_mean" : '
                                             '303386.500000,\n'
                                             '        "iops_stddev" : '
                                             '425.766413,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 30.957143,\n'
                                             '      "sys_cpu" : 69.014286,\n'
                                             '      "ctx" : 24,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999976,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.029906,\n'
                                             '        "20" : 2.873694,\n'
                                             '        "50" : 96.944752,\n'
                                             '        "100" : 0.133187,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=143723: Wed '
                                             'Oct 26 04:49:47 2022\n'
                                             '  read: IOPS=303k, BW=148MiB/s '
                                             '(155MB/s)(1037MiB/7001msec)\n'
                                             '    slat (nsec): min=1523, '
                                             'max=35066, avg=2799.30, '
                                             'stdev=1164.54\n'
                                             '    clat (nsec): min=200, '
                                             'max=131516, avg=23335.46, '
                                             'stdev=2973.49\n'
                                             '     lat (usec): min=6, max=135, '
                                             'avg=26.13, stdev= 2.90\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16768],  '
                                             '5.00th=[21376], 10.00th=[21632], '
                                             '20.00th=[21888],\n'
                                             '     | 30.00th=[22656], '
                                             '40.00th=[22912], '
                                             '50.00th=[23168], '
                                             '60.00th=[23168],\n'
                                             '     | 70.00th=[23680], '
                                             '80.00th=[24192], '
                                             '90.00th=[24448], '
                                             '95.00th=[27008],\n'
                                             '     | 99.00th=[34560], '
                                             '99.50th=[39680], '
                                             '99.90th=[54528], '
                                             '99.95th=[64768],\n'
                                             '     | 99.99th=[76288]\n'
                                             '   bw (  KiB/s): min=151360, '
                                             'max=152173, per=100.00%, '
                                             'avg=151693.14, stdev=212.96, '
                                             'samples=14\n'
                                             '   iops        : min=302720, '
                                             'max=304346, avg=303386.50, '
                                             'stdev=425.77, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.03%, 20=2.87%, 50=96.94%, '
                                             '100=0.13%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=30.96%, '
                                             'sys=69.01%, ctx=24, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2123330,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=148MiB/s (155MB/s), '
                                             '148MiB/s-148MiB/s '
                                             '(155MB/s-155MB/s), io=1037MiB '
                                             '(1087MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.output'),
                                   'state': {'begin': 1666745498.8982599,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7933619022369385,
                                             'end': 1666745499.6916218,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_348.state')},
                       'cmd_349': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.output'),
                                   'state': {'begin': 1666745501.7145822,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7818739414215088,
                                             'end': 1666745502.4964561,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_349.state')},
                       'cmd_35': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.output'),
                                  'state': {'begin': 1666743994.2055538,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-reference_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.037194967269897,
                                            'end': 1666744005.2427487,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_35.state')},
                       'cmd_350': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.output'),
                                   'state': {'begin': 1666745502.498419,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.457935333251953,
                                             'end': 1666745513.9563544,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_350.state')},
                       'cmd_351': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752603,\n'
                                             '  "timestamp_ms" : '
                                             '1666752603063,\n'
                                             '  "time" : "Wed Oct 26 04:50:03 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1081744896,\n'
                                             '        "io_kbytes" : 1056391,\n'
                                             '        "bw_bytes" : 154512911,\n'
                                             '        "bw" : 150891,\n'
                                             '        "iops" : 301782.031138,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2112776,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1522,\n'
                                             '          "max" : 27091,\n'
                                             '          "mean" : 2821.886822,\n'
                                             '          "stddev" : '
                                             '1220.912718,\n'
                                             '          "N" : 2112776\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 190,\n'
                                             '          "max" : 120736,\n'
                                             '          "mean" : '
                                             '23444.546176,\n'
                                             '          "stddev" : '
                                             '2978.404735,\n'
                                             '          "N" : 2112783,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16768,\n'
                                             '            "5.000000" : 21376,\n'
                                             '            "10.000000" : '
                                             '21632,\n'
                                             '            "20.000000" : '
                                             '21888,\n'
                                             '            "30.000000" : '
                                             '22656,\n'
                                             '            "40.000000" : '
                                             '22912,\n'
                                             '            "50.000000" : '
                                             '23168,\n'
                                             '            "60.000000" : '
                                             '23424,\n'
                                             '            "70.000000" : '
                                             '23680,\n'
                                             '            "80.000000" : '
                                             '24192,\n'
                                             '            "90.000000" : '
                                             '24704,\n'
                                             '            "95.000000" : '
                                             '27008,\n'
                                             '            "99.000000" : '
                                             '35072,\n'
                                             '            "99.500000" : '
                                             '39680,\n'
                                             '            "99.900000" : '
                                             '51968,\n'
                                             '            "99.950000" : '
                                             '62208,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6622,\n'
                                             '          "max" : 122389,\n'
                                             '          "mean" : '
                                             '26266.430108,\n'
                                             '          "stddev" : '
                                             '2931.359656,\n'
                                             '          "N" : 2112783\n'
                                             '        },\n'
                                             '        "bw_min" : 150430,\n'
                                             '        "bw_max" : 151161,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '150901.538462,\n'
                                             '        "bw_dev" : 191.142013,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 300860,\n'
                                             '        "iops_max" : 302322,\n'
                                             '        "iops_mean" : '
                                             '301803.076923,\n'
                                             '        "iops_stddev" : '
                                             '382.467093,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 31.242857,\n'
                                             '      "sys_cpu" : 68.728571,\n'
                                             '      "ctx" : 25,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999976,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.029440,\n'
                                             '        "20" : 2.844267,\n'
                                             '        "50" : 96.992298,\n'
                                             '        "100" : 0.116103,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=143773: Wed '
                                             'Oct 26 04:50:03 2022\n'
                                             '  read: IOPS=302k, BW=147MiB/s '
                                             '(155MB/s)(1032MiB/7001msec)\n'
                                             '    slat (nsec): min=1522, '
                                             'max=27091, avg=2821.89, '
                                             'stdev=1220.91\n'
                                             '    clat (nsec): min=190, '
                                             'max=120736, avg=23444.55, '
                                             'stdev=2978.40\n'
                                             '     lat (usec): min=6, max=122, '
                                             'avg=26.27, stdev= 2.93\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16768],  '
                                             '5.00th=[21376], 10.00th=[21632], '
                                             '20.00th=[21888],\n'
                                             '     | 30.00th=[22656], '
                                             '40.00th=[22912], '
                                             '50.00th=[23168], '
                                             '60.00th=[23424],\n'
                                             '     | 70.00th=[23680], '
                                             '80.00th=[24192], '
                                             '90.00th=[24704], '
                                             '95.00th=[27008],\n'
                                             '     | 99.00th=[35072], '
                                             '99.50th=[39680], '
                                             '99.90th=[51968], '
                                             '99.95th=[62208],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=150430, '
                                             'max=151161, per=100.00%, '
                                             'avg=150901.54, stdev=191.14, '
                                             'samples=13\n'
                                             '   iops        : min=300860, '
                                             'max=302322, avg=301803.08, '
                                             'stdev=382.47, samples=13\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.03%, 20=2.84%, 50=96.99%, '
                                             '100=0.12%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=31.24%, '
                                             'sys=68.73%, ctx=25, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2112776,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=147MiB/s (155MB/s), '
                                             '147MiB/s-147MiB/s '
                                             '(155MB/s-155MB/s), io=1032MiB '
                                             '(1082MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.output'),
                                   'state': {'begin': 1666745513.9593008,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7880733013153076,
                                             'end': 1666745514.747374,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_351.state')},
                       'cmd_352': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.output'),
                                   'state': {'begin': 1666745515.7417245,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7850258350372314,
                                             'end': 1666745516.5267503,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_352.state')},
                       'cmd_353': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.output'),
                                   'state': {'begin': 1666745516.5291083,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.059075117111206,
                                             'end': 1666745527.5881834,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_353.state')},
                       'cmd_354': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752616,\n'
                                             '  "timestamp_ms" : '
                                             '1666752616765,\n'
                                             '  "time" : "Wed Oct 26 04:50:16 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1090587136,\n'
                                             '        "io_kbytes" : 1065026,\n'
                                             '        "bw_bytes" : 155775908,\n'
                                             '        "bw" : 152124,\n'
                                             '        "iops" : 304248.678760,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2130045,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1382,\n'
                                             '          "max" : 75892,\n'
                                             '          "mean" : 2807.293279,\n'
                                             '          "stddev" : '
                                             '1237.975793,\n'
                                             '          "N" : 2130046\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 140474,\n'
                                             '          "mean" : '
                                             '23243.132478,\n'
                                             '          "stddev" : '
                                             '4091.818656,\n'
                                             '          "N" : 2130052,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 15424,\n'
                                             '            "5.000000" : 19328,\n'
                                             '            "10.000000" : '
                                             '19584,\n'
                                             '            "20.000000" : '
                                             '21120,\n'
                                             '            "30.000000" : '
                                             '21632,\n'
                                             '            "40.000000" : '
                                             '22656,\n'
                                             '            "50.000000" : '
                                             '22912,\n'
                                             '            "60.000000" : '
                                             '23168,\n'
                                             '            "70.000000" : '
                                             '23936,\n'
                                             '            "80.000000" : '
                                             '24704,\n'
                                             '            "90.000000" : '
                                             '25984,\n'
                                             '            "95.000000" : '
                                             '27264,\n'
                                             '            "99.000000" : '
                                             '43264,\n'
                                             '            "99.500000" : '
                                             '45824,\n'
                                             '            "99.900000" : '
                                             '56576,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 81408\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6923,\n'
                                             '          "max" : 141916,\n'
                                             '          "mean" : '
                                             '26050.426103,\n'
                                             '          "stddev" : '
                                             '4239.486761,\n'
                                             '          "N" : 2130052\n'
                                             '        },\n'
                                             '        "bw_min" : 151886,\n'
                                             '        "bw_max" : 152476,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '152167.357143,\n'
                                             '        "bw_dev" : 193.514304,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 303772,\n'
                                             '        "iops_max" : 304952,\n'
                                             '        "iops_mean" : '
                                             '304334.714286,\n'
                                             '        "iops_stddev" : '
                                             '386.864800,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.885714,\n'
                                             '      "sys_cpu" : 80.342857,\n'
                                             '      "ctx" : 11485,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999977,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.044882,\n'
                                             '        "20" : 15.144750,\n'
                                             '        "50" : 84.647883,\n'
                                             '        "100" : 0.147649,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=143812: Wed '
                                             'Oct 26 04:50:16 2022\n'
                                             '  read: IOPS=304k, BW=149MiB/s '
                                             '(156MB/s)(1040MiB/7001msec)\n'
                                             '    slat (nsec): min=1382, '
                                             'max=75892, avg=2807.29, '
                                             'stdev=1237.98\n'
                                             '    clat (nsec): min=110, '
                                             'max=140474, avg=23243.13, '
                                             'stdev=4091.82\n'
                                             '     lat (usec): min=6, max=141, '
                                             'avg=26.05, stdev= 4.24\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[15424],  '
                                             '5.00th=[19328], 10.00th=[19584], '
                                             '20.00th=[21120],\n'
                                             '     | 30.00th=[21632], '
                                             '40.00th=[22656], '
                                             '50.00th=[22912], '
                                             '60.00th=[23168],\n'
                                             '     | 70.00th=[23936], '
                                             '80.00th=[24704], '
                                             '90.00th=[25984], '
                                             '95.00th=[27264],\n'
                                             '     | 99.00th=[43264], '
                                             '99.50th=[45824], '
                                             '99.90th=[56576], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[81408]\n'
                                             '   bw (  KiB/s): min=151886, '
                                             'max=152476, per=100.00%, '
                                             'avg=152167.36, stdev=193.51, '
                                             'samples=14\n'
                                             '   iops        : min=303772, '
                                             'max=304952, avg=304334.71, '
                                             'stdev=386.86, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.04%, 20=15.14%, '
                                             '50=84.65%\n'
                                             '  lat (usec)   : 100=0.15%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=17.89%, '
                                             'sys=80.34%, ctx=11485, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2130045,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=149MiB/s (156MB/s), '
                                             '149MiB/s-149MiB/s '
                                             '(156MB/s-156MB/s), io=1040MiB '
                                             '(1091MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.output'),
                                   'state': {'begin': 1666745527.5914118,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7916409969329834,
                                             'end': 1666745528.3830528,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_354.state')},
                       'cmd_355': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.output'),
                                   'state': {'begin': 1666745529.3818274,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7902827262878418,
                                             'end': 1666745530.17211,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_355.state')},
                       'cmd_356': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.output'),
                                   'state': {'begin': 1666745530.1745954,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.050925016403198,
                                             'end': 1666745541.2255204,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_356.state')},
                       'cmd_357': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752630,\n'
                                             '  "timestamp_ms" : '
                                             '1666752630403,\n'
                                             '  "time" : "Wed Oct 26 04:50:30 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1091710976,\n'
                                             '        "io_kbytes" : 1066124,\n'
                                             '        "bw_bytes" : 155936434,\n'
                                             '        "bw" : 152281,\n'
                                             '        "iops" : 304562.205399,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2132240,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1372,\n'
                                             '          "max" : 65753,\n'
                                             '          "mean" : 2806.080467,\n'
                                             '          "stddev" : '
                                             '1233.481209,\n'
                                             '          "N" : 2132241\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 142047,\n'
                                             '          "mean" : '
                                             '23217.680802,\n'
                                             '          "stddev" : '
                                             '4093.965853,\n'
                                             '          "N" : 2132247,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 15424,\n'
                                             '            "5.000000" : 19328,\n'
                                             '            "10.000000" : '
                                             '19584,\n'
                                             '            "20.000000" : '
                                             '21120,\n'
                                             '            "30.000000" : '
                                             '21632,\n'
                                             '            "40.000000" : '
                                             '22656,\n'
                                             '            "50.000000" : '
                                             '22912,\n'
                                             '            "60.000000" : '
                                             '23168,\n'
                                             '            "70.000000" : '
                                             '23936,\n'
                                             '            "80.000000" : '
                                             '24704,\n'
                                             '            "90.000000" : '
                                             '25984,\n'
                                             '            "95.000000" : '
                                             '27264,\n'
                                             '            "99.000000" : '
                                             '43264,\n'
                                             '            "99.500000" : '
                                             '45824,\n'
                                             '            "99.900000" : '
                                             '57600,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 81408\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6883,\n'
                                             '          "max" : 143439,\n'
                                             '          "mean" : '
                                             '26023.761436,\n'
                                             '          "stddev" : '
                                             '4231.145211,\n'
                                             '          "N" : 2132247\n'
                                             '        },\n'
                                             '        "bw_min" : 152213,\n'
                                             '        "bw_max" : 152540,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '152322.357143,\n'
                                             '        "bw_dev" : 94.171372,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 304426,\n'
                                             '        "iops_max" : 305080,\n'
                                             '        "iops_mean" : '
                                             '304644.857143,\n'
                                             '        "iops_stddev" : '
                                             '188.408254,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.142857,\n'
                                             '      "sys_cpu" : 79.042857,\n'
                                             '      "ctx" : 11717,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999977,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.046196,\n'
                                             '        "20" : 14.669643,\n'
                                             '        "50" : 85.119499,\n'
                                             '        "100" : 0.149655,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=143851: Wed '
                                             'Oct 26 04:50:30 2022\n'
                                             '  read: IOPS=305k, BW=149MiB/s '
                                             '(156MB/s)(1041MiB/7001msec)\n'
                                             '    slat (nsec): min=1372, '
                                             'max=65753, avg=2806.08, '
                                             'stdev=1233.48\n'
                                             '    clat (nsec): min=110, '
                                             'max=142047, avg=23217.68, '
                                             'stdev=4093.97\n'
                                             '     lat (usec): min=6, max=143, '
                                             'avg=26.02, stdev= 4.23\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[15424],  '
                                             '5.00th=[19328], 10.00th=[19584], '
                                             '20.00th=[21120],\n'
                                             '     | 30.00th=[21632], '
                                             '40.00th=[22656], '
                                             '50.00th=[22912], '
                                             '60.00th=[23168],\n'
                                             '     | 70.00th=[23936], '
                                             '80.00th=[24704], '
                                             '90.00th=[25984], '
                                             '95.00th=[27264],\n'
                                             '     | 99.00th=[43264], '
                                             '99.50th=[45824], '
                                             '99.90th=[57600], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[81408]\n'
                                             '   bw (  KiB/s): min=152213, '
                                             'max=152540, per=100.00%, '
                                             'avg=152322.36, stdev=94.17, '
                                             'samples=14\n'
                                             '   iops        : min=304426, '
                                             'max=305080, avg=304644.86, '
                                             'stdev=188.41, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.05%, 20=14.67%, '
                                             '50=85.12%\n'
                                             '  lat (usec)   : 100=0.15%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=19.14%, '
                                             'sys=79.04%, ctx=11717, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2132240,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=149MiB/s (156MB/s), '
                                             '149MiB/s-149MiB/s '
                                             '(156MB/s-156MB/s), io=1041MiB '
                                             '(1092MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.output'),
                                   'state': {'begin': 1666745541.2280989,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7909369468688965,
                                             'end': 1666745542.0190358,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_357.state')},
                       'cmd_358': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.output'),
                                   'state': {'begin': 1666745543.012986,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7926218509674072,
                                             'end': 1666745543.8056078,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_358.state')},
                       'cmd_359': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.output'),
                                   'state': {'begin': 1666745543.807617,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_io_uring-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.05995512008667,
                                             'end': 1666745554.867572,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_359.state')},
                       'cmd_36': {'output': 'io_uring-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=io_uring, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751094,\n'
                                            '  "timestamp_ms" : '
                                            '1666751094444,\n'
                                            '  "time" : "Wed Oct 26 04:24:54 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "io_uring",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 286309376,\n'
                                            '        "io_kbytes" : 279599,\n'
                                            '        "bw_bytes" : 40895497,\n'
                                            '        "bw" : 39937,\n'
                                            '        "iops" : 79874.017997,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 559198,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2494,\n'
                                            '          "max" : 45485,\n'
                                            '          "mean" : 2642.445052,\n'
                                            '          "stddev" : 228.167226,\n'
                                            '          "N" : 559198\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 190,\n'
                                            '          "max" : 81473,\n'
                                            '          "mean" : 9223.417267,\n'
                                            '          "stddev" : '
                                            '1276.987473,\n'
                                            '          "N" : 559198,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8768,\n'
                                            '            "5.000000" : 8896,\n'
                                            '            "10.000000" : 8896,\n'
                                            '            "20.000000" : 9024,\n'
                                            '            "30.000000" : 9024,\n'
                                            '            "40.000000" : 9024,\n'
                                            '            "50.000000" : 9024,\n'
                                            '            "60.000000" : 9024,\n'
                                            '            "70.000000" : 9152,\n'
                                            '            "80.000000" : 9152,\n'
                                            '            "90.000000" : 9152,\n'
                                            '            "95.000000" : 9408,\n'
                                            '            "99.000000" : 15424,\n'
                                            '            "99.500000" : 16192,\n'
                                            '            "99.900000" : 19584,\n'
                                            '            "99.950000" : 23168,\n'
                                            '            "99.990000" : 56576\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 8957,\n'
                                            '          "max" : 84108,\n'
                                            '          "mean" : 11865.862319,\n'
                                            '          "stddev" : '
                                            '1296.542780,\n'
                                            '          "N" : 559198\n'
                                            '        },\n'
                                            '        "bw_min" : 39762,\n'
                                            '        "bw_max" : 40171,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '39954.785714,\n'
                                            '        "bw_dev" : 110.705554,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 79524,\n'
                                            '        "iops_max" : 80342,\n'
                                            '        "iops_mean" : '
                                            '79909.642857,\n'
                                            '        "iops_stddev" : '
                                            '221.379454,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 14.328571,\n'
                                            '      "sys_cpu" : 46.328571,\n'
                                            '      "ctx" : 559054,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 97.150741,\n'
                                            '        "20" : 2.758057,\n'
                                            '        "50" : 0.062590,\n'
                                            '        "100" : 0.018062,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 801785,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5027,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5027,\n'
                                            '      "util" : 99.098564\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=139136: Wed '
                                            'Oct 26 04:24:54 2022\n'
                                            '  read: IOPS=79.9k, BW=39.0MiB/s '
                                            '(40.9MB/s)(273MiB/7001msec)\n'
                                            '    slat (nsec): min=2494, '
                                            'max=45485, avg=2642.45, '
                                            'stdev=228.17\n'
                                            '    clat (nsec): min=190, '
                                            'max=81473, avg=9223.42, '
                                            'stdev=1276.99\n'
                                            '     lat (nsec): min=8957, '
                                            'max=84108, avg=11865.86, '
                                            'stdev=1296.54\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8768],  5.00th=[ '
                                            '8896], 10.00th=[ 8896], 20.00th=[ '
                                            '9024],\n'
                                            '     | 30.00th=[ 9024], 40.00th=[ '
                                            '9024], 50.00th=[ 9024], 60.00th=[ '
                                            '9024],\n'
                                            '     | 70.00th=[ 9152], 80.00th=[ '
                                            '9152], 90.00th=[ 9152], 95.00th=[ '
                                            '9408],\n'
                                            '     | 99.00th=[15424], '
                                            '99.50th=[16192], 99.90th=[19584], '
                                            '99.95th=[23168],\n'
                                            '     | 99.99th=[56576]\n'
                                            '   bw (  KiB/s): min=39762, '
                                            'max=40171, per=100.00%, '
                                            'avg=39954.79, stdev=110.71, '
                                            'samples=14\n'
                                            '   iops        : min=79524, '
                                            'max=80342, avg=79909.64, '
                                            'stdev=221.38, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 10=97.15%, '
                                            '20=2.76%, 50=0.06%, 100=0.02%\n'
                                            '  cpu          : usr=14.33%, '
                                            'sys=46.33%, ctx=559054, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=559198,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=39.0MiB/s (40.9MB/s), '
                                            '39.0MiB/s-39.0MiB/s '
                                            '(40.9MB/s-40.9MB/s), io=273MiB '
                                            '(286MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=801785/0, '
                                            'merge=0/0, ticks=5027/0, '
                                            'in_queue=5027, util=99.10%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.output'),
                                  'state': {'begin': 1666744005.2492862,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7900209426879883,
                                            'end': 1666744006.039307,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_36.state')},
                       'cmd_360': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752644,\n'
                                             '  "timestamp_ms" : '
                                             '1666752644045,\n'
                                             '  "time" : "Wed Oct 26 04:50:44 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1097063424,\n'
                                             '        "io_kbytes" : 1071351,\n'
                                             '        "bw_bytes" : 156700960,\n'
                                             '        "bw" : 153028,\n'
                                             '        "iops" : 306055.420654,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2142694,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1362,\n'
                                             '          "max" : 43081,\n'
                                             '          "mean" : 2770.524205,\n'
                                             '          "stddev" : '
                                             '1257.270481,\n'
                                             '          "N" : 2142695\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 142568,\n'
                                             '          "mean" : '
                                             '23118.608042,\n'
                                             '          "stddev" : '
                                             '3988.582641,\n'
                                             '          "N" : 2142701,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 15424,\n'
                                             '            "5.000000" : 19328,\n'
                                             '            "10.000000" : '
                                             '20864,\n'
                                             '            "20.000000" : '
                                             '21120,\n'
                                             '            "30.000000" : '
                                             '21376,\n'
                                             '            "40.000000" : '
                                             '22400,\n'
                                             '            "50.000000" : '
                                             '22656,\n'
                                             '            "60.000000" : '
                                             '22912,\n'
                                             '            "70.000000" : '
                                             '23680,\n'
                                             '            "80.000000" : '
                                             '24192,\n'
                                             '            "90.000000" : '
                                             '25472,\n'
                                             '            "95.000000" : '
                                             '27776,\n'
                                             '            "99.000000" : '
                                             '42752,\n'
                                             '            "99.500000" : '
                                             '45824,\n'
                                             '            "99.900000" : '
                                             '58624,\n'
                                             '            "99.950000" : '
                                             '68096,\n'
                                             '            "99.990000" : 82432\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6923,\n'
                                             '          "max" : 143950,\n'
                                             '          "mean" : '
                                             '25889.131866,\n'
                                             '          "stddev" : '
                                             '4007.357932,\n'
                                             '          "N" : 2142701\n'
                                             '        },\n'
                                             '        "bw_min" : 152809,\n'
                                             '        "bw_max" : 153283,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '153068.142857,\n'
                                             '        "bw_dev" : 168.665740,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 305618,\n'
                                             '        "iops_max" : 306566,\n'
                                             '        "iops_mean" : '
                                             '306136.214286,\n'
                                             '        "iops_stddev" : '
                                             '337.609738,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.928571,\n'
                                             '      "sys_cpu" : 78.200000,\n'
                                             '      "ctx" : 12004,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999977,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.040043,\n'
                                             '        "20" : 8.114318,\n'
                                             '        "50" : 91.677533,\n'
                                             '        "100" : 0.156905,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=143892: Wed '
                                             'Oct 26 04:50:44 2022\n'
                                             '  read: IOPS=306k, BW=149MiB/s '
                                             '(157MB/s)(1046MiB/7001msec)\n'
                                             '    slat (nsec): min=1362, '
                                             'max=43081, avg=2770.52, '
                                             'stdev=1257.27\n'
                                             '    clat (nsec): min=110, '
                                             'max=142568, avg=23118.61, '
                                             'stdev=3988.58\n'
                                             '     lat (usec): min=6, max=143, '
                                             'avg=25.89, stdev= 4.01\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[15424],  '
                                             '5.00th=[19328], 10.00th=[20864], '
                                             '20.00th=[21120],\n'
                                             '     | 30.00th=[21376], '
                                             '40.00th=[22400], '
                                             '50.00th=[22656], '
                                             '60.00th=[22912],\n'
                                             '     | 70.00th=[23680], '
                                             '80.00th=[24192], '
                                             '90.00th=[25472], '
                                             '95.00th=[27776],\n'
                                             '     | 99.00th=[42752], '
                                             '99.50th=[45824], '
                                             '99.90th=[58624], '
                                             '99.95th=[68096],\n'
                                             '     | 99.99th=[82432]\n'
                                             '   bw (  KiB/s): min=152809, '
                                             'max=153283, per=100.00%, '
                                             'avg=153068.14, stdev=168.67, '
                                             'samples=14\n'
                                             '   iops        : min=305618, '
                                             'max=306566, avg=306136.21, '
                                             'stdev=337.61, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.04%, 20=8.11%, '
                                             '50=91.68%\n'
                                             '  lat (usec)   : 100=0.16%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=19.93%, '
                                             'sys=78.20%, ctx=12004, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2142694,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=149MiB/s (157MB/s), '
                                             '149MiB/s-149MiB/s '
                                             '(157MB/s-157MB/s), io=1046MiB '
                                             '(1097MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.output'),
                                   'state': {'begin': 1666745554.8706086,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7866778373718262,
                                             'end': 1666745555.6572864,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_360.state')},
                       'cmd_361': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.output'),
                                   'state': {'begin': 1666745556.657499,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8002305030822754,
                                             'end': 1666745557.4577296,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_361.state')},
                       'cmd_362': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.output'),
                                   'state': {'begin': 1666745557.45975,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-reference_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.076076030731201,
                                             'end': 1666745568.535826,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_362.state')},
                       'cmd_363': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752657,\n'
                                             '  "timestamp_ms" : '
                                             '1666752657713,\n'
                                             '  "time" : "Wed Oct 26 04:50:57 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1046272512,\n'
                                             '        "io_kbytes" : 1021750,\n'
                                             '        "bw_bytes" : 149446152,\n'
                                             '        "bw" : 145943,\n'
                                             '        "iops" : 291885.873447,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2043493,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1402,\n'
                                             '          "max" : 113964,\n'
                                             '          "mean" : 2707.785157,\n'
                                             '          "stddev" : '
                                             '1337.154939,\n'
                                             '          "N" : 2043494\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2595,\n'
                                             '          "max" : 138590,\n'
                                             '          "mean" : '
                                             '24451.180669,\n'
                                             '          "stddev" : '
                                             '3177.840339,\n'
                                             '          "N" : 2043500,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 17536,\n'
                                             '            "5.000000" : 22400,\n'
                                             '            "10.000000" : '
                                             '22656,\n'
                                             '            "20.000000" : '
                                             '22656,\n'
                                             '            "30.000000" : '
                                             '22912,\n'
                                             '            "40.000000" : '
                                             '23680,\n'
                                             '            "50.000000" : '
                                             '24192,\n'
                                             '            "60.000000" : '
                                             '24448,\n'
                                             '            "70.000000" : '
                                             '25216,\n'
                                             '            "80.000000" : '
                                             '25472,\n'
                                             '            "90.000000" : '
                                             '25728,\n'
                                             '            "95.000000" : '
                                             '29312,\n'
                                             '            "99.000000" : '
                                             '37632,\n'
                                             '            "99.500000" : '
                                             '42240,\n'
                                             '            "99.900000" : '
                                             '54528,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10450,\n'
                                             '          "max" : 143930,\n'
                                             '          "mean" : '
                                             '27158.965725,\n'
                                             '          "stddev" : '
                                             '3022.018116,\n'
                                             '          "N" : 2043500\n'
                                             '        },\n'
                                             '        "bw_min" : 145819,\n'
                                             '        "bw_max" : 146239,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '146001.692308,\n'
                                             '        "bw_dev" : 141.640028,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 291638,\n'
                                             '        "iops_max" : 292478,\n'
                                             '        "iops_mean" : '
                                             '292003.461538,\n'
                                             '        "iops_stddev" : '
                                             '283.176216,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 35.971429,\n'
                                             '      "sys_cpu" : 62.771429,\n'
                                             '      "ctx" : 10182,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999951,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 2.033870,\n'
                                             '        "50" : 97.828033,\n'
                                             '        "100" : 0.127282,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2858731,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 15931,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 15931,\n'
                                             '      "util" : 99.035239\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=143932: Wed '
                                             'Oct 26 04:50:57 2022\n'
                                             '  read: IOPS=292k, BW=143MiB/s '
                                             '(149MB/s)(998MiB/7001msec)\n'
                                             '    slat (nsec): min=1402, '
                                             'max=113964, avg=2707.79, '
                                             'stdev=1337.15\n'
                                             '    clat (usec): min=2, max=138, '
                                             'avg=24.45, stdev= 3.18\n'
                                             '     lat (usec): min=10, '
                                             'max=143, avg=27.16, stdev= 3.02\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[17536],  '
                                             '5.00th=[22400], 10.00th=[22656], '
                                             '20.00th=[22656],\n'
                                             '     | 30.00th=[22912], '
                                             '40.00th=[23680], '
                                             '50.00th=[24192], '
                                             '60.00th=[24448],\n'
                                             '     | 70.00th=[25216], '
                                             '80.00th=[25472], '
                                             '90.00th=[25728], '
                                             '95.00th=[29312],\n'
                                             '     | 99.00th=[37632], '
                                             '99.50th=[42240], '
                                             '99.90th=[54528], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=145819, '
                                             'max=146239, per=100.00%, '
                                             'avg=146001.69, stdev=141.64, '
                                             'samples=13\n'
                                             '   iops        : min=291638, '
                                             'max=292478, avg=292003.46, '
                                             'stdev=283.18, samples=13\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=2.03%, 50=97.83%, '
                                             '100=0.13%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=35.97%, '
                                             'sys=62.77%, ctx=10182, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2043493,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=143MiB/s (149MB/s), '
                                             '143MiB/s-143MiB/s '
                                             '(149MB/s-149MB/s), io=998MiB '
                                             '(1046MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2858731/0, '
                                             'merge=0/0, ticks=15931/0, '
                                             'in_queue=15931, util=99.04%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.output'),
                                   'state': {'begin': 1666745568.538666,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7850360870361328,
                                             'end': 1666745569.323702,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_363.state')},
                       'cmd_364': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.output'),
                                   'state': {'begin': 1666745570.3126845,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7866883277893066,
                                             'end': 1666745571.0993729,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_364.state')},
                       'cmd_365': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.output'),
                                   'state': {'begin': 1666745571.1018023,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-reference_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.070146083831787,
                                             'end': 1666745582.1719484,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_365.state')},
                       'cmd_366': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752671,\n'
                                             '  "timestamp_ms" : '
                                             '1666752671349,\n'
                                             '  "time" : "Wed Oct 26 04:51:11 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1046752768,\n'
                                             '        "io_kbytes" : 1022219,\n'
                                             '        "bw_bytes" : 149514750,\n'
                                             '        "bw" : 146010,\n'
                                             '        "iops" : 292019.854307,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2044431,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1403,\n'
                                             '          "max" : 123251,\n'
                                             '          "mean" : 2715.633511,\n'
                                             '          "stddev" : '
                                             '1330.313775,\n'
                                             '          "N" : 2044432\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2605,\n'
                                             '          "max" : 145643,\n'
                                             '          "mean" : '
                                             '24432.144750,\n'
                                             '          "stddev" : '
                                             '3169.438036,\n'
                                             '          "N" : 2044438,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 17536,\n'
                                             '            "5.000000" : 22400,\n'
                                             '            "10.000000" : '
                                             '22656,\n'
                                             '            "20.000000" : '
                                             '22656,\n'
                                             '            "30.000000" : '
                                             '22912,\n'
                                             '            "40.000000" : '
                                             '23680,\n'
                                             '            "50.000000" : '
                                             '23936,\n'
                                             '            "60.000000" : '
                                             '24448,\n'
                                             '            "70.000000" : '
                                             '24960,\n'
                                             '            "80.000000" : '
                                             '25472,\n'
                                             '            "90.000000" : '
                                             '25728,\n'
                                             '            "95.000000" : '
                                             '29312,\n'
                                             '            "99.000000" : '
                                             '37632,\n'
                                             '            "99.500000" : '
                                             '41728,\n'
                                             '            "99.900000" : '
                                             '52992,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10781,\n'
                                             '          "max" : 148709,\n'
                                             '          "mean" : '
                                             '27147.777613,\n'
                                             '          "stddev" : '
                                             '3031.587372,\n'
                                             '          "N" : 2044438\n'
                                             '        },\n'
                                             '        "bw_min" : 145798,\n'
                                             '        "bw_max" : 146766,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '146077.000000,\n'
                                             '        "bw_dev" : 239.693394,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 291596,\n'
                                             '        "iops_max" : 293532,\n'
                                             '        "iops_mean" : '
                                             '292154.142857,\n'
                                             '        "iops_stddev" : '
                                             '479.286466,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 38.614286,\n'
                                             '      "sys_cpu" : 60.157143,\n'
                                             '      "ctx" : 10093,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999951,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 2.060720,\n'
                                             '        "50" : 97.807507,\n'
                                             '        "100" : 0.119887,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2933310,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 16349,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 16349,\n'
                                             '      "util" : 99.098564\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=143970: Wed '
                                             'Oct 26 04:51:11 2022\n'
                                             '  read: IOPS=292k, BW=143MiB/s '
                                             '(150MB/s)(998MiB/7001msec)\n'
                                             '    slat (nsec): min=1403, '
                                             'max=123251, avg=2715.63, '
                                             'stdev=1330.31\n'
                                             '    clat (usec): min=2, max=145, '
                                             'avg=24.43, stdev= 3.17\n'
                                             '     lat (usec): min=10, '
                                             'max=148, avg=27.15, stdev= 3.03\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[17536],  '
                                             '5.00th=[22400], 10.00th=[22656], '
                                             '20.00th=[22656],\n'
                                             '     | 30.00th=[22912], '
                                             '40.00th=[23680], '
                                             '50.00th=[23936], '
                                             '60.00th=[24448],\n'
                                             '     | 70.00th=[24960], '
                                             '80.00th=[25472], '
                                             '90.00th=[25728], '
                                             '95.00th=[29312],\n'
                                             '     | 99.00th=[37632], '
                                             '99.50th=[41728], '
                                             '99.90th=[52992], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=145798, '
                                             'max=146766, per=100.00%, '
                                             'avg=146077.00, stdev=239.69, '
                                             'samples=14\n'
                                             '   iops        : min=291596, '
                                             'max=293532, avg=292154.14, '
                                             'stdev=479.29, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=2.06%, 50=97.81%, '
                                             '100=0.12%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=38.61%, '
                                             'sys=60.16%, ctx=10093, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2044431,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=143MiB/s (150MB/s), '
                                             '143MiB/s-143MiB/s '
                                             '(150MB/s-150MB/s), io=998MiB '
                                             '(1047MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2933310/0, '
                                             'merge=0/0, ticks=16349/0, '
                                             'in_queue=16349, util=99.10%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.output'),
                                   'state': {'begin': 1666745582.1785822,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.791018009185791,
                                             'end': 1666745582.9696002,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_366.state')},
                       'cmd_367': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.output'),
                                   'state': {'begin': 1666745583.9647493,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7866001129150391,
                                             'end': 1666745584.7513494,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_367.state')},
                       'cmd_368': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.output'),
                                   'state': {'begin': 1666745584.753581,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-reference_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.070449590682983,
                                             'end': 1666745595.8240306,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_368.state')},
                       'cmd_369': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=libaio, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752685,\n'
                                             '  "timestamp_ms" : '
                                             '1666752685000,\n'
                                             '  "time" : "Wed Oct 26 04:51:25 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1043872256,\n'
                                             '        "io_kbytes" : 1019406,\n'
                                             '        "bw_bytes" : 149103307,\n'
                                             '        "bw" : 145608,\n'
                                             '        "iops" : 291216.254821,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2038805,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1432,\n'
                                             '          "max" : 27141,\n'
                                             '          "mean" : 2732.441553,\n'
                                             '          "stddev" : '
                                             '1335.007083,\n'
                                             '          "N" : 2038806\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4679,\n'
                                             '          "max" : 159368,\n'
                                             '          "mean" : '
                                             '24492.867077,\n'
                                             '          "stddev" : '
                                             '3138.428213,\n'
                                             '          "N" : 2038812,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 17536,\n'
                                             '            "5.000000" : 22400,\n'
                                             '            "10.000000" : '
                                             '22656,\n'
                                             '            "20.000000" : '
                                             '22912,\n'
                                             '            "30.000000" : '
                                             '22912,\n'
                                             '            "40.000000" : '
                                             '23680,\n'
                                             '            "50.000000" : '
                                             '24192,\n'
                                             '            "60.000000" : '
                                             '24960,\n'
                                             '            "70.000000" : '
                                             '25216,\n'
                                             '            "80.000000" : '
                                             '25472,\n'
                                             '            "90.000000" : '
                                             '25728,\n'
                                             '            "95.000000" : '
                                             '29312,\n'
                                             '            "99.000000" : '
                                             '37632,\n'
                                             '            "99.500000" : '
                                             '41728,\n'
                                             '            "99.900000" : '
                                             '52480,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 78336\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10911,\n'
                                             '          "max" : 160841,\n'
                                             '          "mean" : '
                                             '27225.311592,\n'
                                             '          "stddev" : '
                                             '2968.109528,\n'
                                             '          "N" : 2038812\n'
                                             '        },\n'
                                             '        "bw_min" : 145239,\n'
                                             '        "bw_max" : 145957,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '145648.230769,\n'
                                             '        "bw_dev" : 185.987972,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 290478,\n'
                                             '        "iops_max" : 291914,\n'
                                             '        "iops_mean" : '
                                             '291296.538462,\n'
                                             '        "iops_stddev" : '
                                             '372.038218,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 37.700000,\n'
                                             '      "sys_cpu" : 61.085714,\n'
                                             '      "ctx" : 10154,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999951,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 1.971253,\n'
                                             '        "50" : 97.900584,\n'
                                             '        "100" : 0.117912,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2855695,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 15886,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 15886,\n'
                                             '      "util" : 99.045298\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=144008: Wed '
                                             'Oct 26 04:51:24 2022\n'
                                             '  read: IOPS=291k, BW=142MiB/s '
                                             '(149MB/s)(996MiB/7001msec)\n'
                                             '    slat (nsec): min=1432, '
                                             'max=27141, avg=2732.44, '
                                             'stdev=1335.01\n'
                                             '    clat (usec): min=4, max=159, '
                                             'avg=24.49, stdev= 3.14\n'
                                             '     lat (usec): min=10, '
                                             'max=160, avg=27.23, stdev= 2.97\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[17536],  '
                                             '5.00th=[22400], 10.00th=[22656], '
                                             '20.00th=[22912],\n'
                                             '     | 30.00th=[22912], '
                                             '40.00th=[23680], '
                                             '50.00th=[24192], '
                                             '60.00th=[24960],\n'
                                             '     | 70.00th=[25216], '
                                             '80.00th=[25472], '
                                             '90.00th=[25728], '
                                             '95.00th=[29312],\n'
                                             '     | 99.00th=[37632], '
                                             '99.50th=[41728], '
                                             '99.90th=[52480], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[78336]\n'
                                             '   bw (  KiB/s): min=145239, '
                                             'max=145957, per=100.00%, '
                                             'avg=145648.23, stdev=185.99, '
                                             'samples=13\n'
                                             '   iops        : min=290478, '
                                             'max=291914, avg=291296.54, '
                                             'stdev=372.04, samples=13\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=1.97%, 50=97.90%, 100=0.12%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=37.70%, '
                                             'sys=61.09%, ctx=10154, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2038805,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=142MiB/s (149MB/s), '
                                             '142MiB/s-142MiB/s '
                                             '(149MB/s-149MB/s), io=996MiB '
                                             '(1044MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2855695/0, '
                                             'merge=0/0, ticks=15886/0, '
                                             'in_queue=15886, util=99.05%\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.output'),
                                   'state': {'begin': 1666745595.8270845,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7876555919647217,
                                             'end': 1666745596.6147401,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_369.state')},
                       'cmd_37': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.output'),
                                  'state': {'begin': 1666744008.0577302,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7847611904144287,
                                            'end': 1666744008.8424914,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_37.state')},
                       'cmd_370': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.output'),
                                   'state': {'begin': 1666745598.6471202,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7935197353363037,
                                             'end': 1666745599.44064,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_370.state')},
                       'cmd_371': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.output'),
                                   'state': {'begin': 1666745599.4430456,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_aio_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.500689506530762,
                                             'end': 1666745610.9437351,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_371.state')},
                       'cmd_372': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752700,\n'
                                             '  "timestamp_ms" : '
                                             '1666752700018,\n'
                                             '  "time" : "Wed Oct 26 04:51:40 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1086418944,\n'
                                             '        "io_kbytes" : 1060956,\n'
                                             '        "bw_bytes" : 155180537,\n'
                                             '        "bw" : 151543,\n'
                                             '        "iops" : 303086.559063,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2121909,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1593,\n'
                                             '          "max" : 35116,\n'
                                             '          "mean" : 2769.186245,\n'
                                             '          "stddev" : '
                                             '1002.279645,\n'
                                             '          "N" : 2121910\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 330,\n'
                                             '          "max" : 188353,\n'
                                             '          "mean" : '
                                             '14078.489138,\n'
                                             '          "stddev" : '
                                             '6683.083761,\n'
                                             '          "N" : 2121912,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3408,\n'
                                             '            "5.000000" : 3536,\n'
                                             '            "10.000000" : 4832,\n'
                                             '            "20.000000" : 6816,\n'
                                             '            "30.000000" : 9664,\n'
                                             '            "40.000000" : '
                                             '12096,\n'
                                             '            "50.000000" : '
                                             '14272,\n'
                                             '            "60.000000" : '
                                             '16320,\n'
                                             '            "70.000000" : '
                                             '18816,\n'
                                             '            "80.000000" : '
                                             '20608,\n'
                                             '            "90.000000" : '
                                             '22912,\n'
                                             '            "95.000000" : '
                                             '23424,\n'
                                             '            "99.000000" : '
                                             '27264,\n'
                                             '            "99.500000" : '
                                             '29824,\n'
                                             '            "99.900000" : '
                                             '42752,\n'
                                             '            "99.950000" : '
                                             '57088,\n'
                                             '            "99.990000" : 70144\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6182,\n'
                                             '          "max" : 191369,\n'
                                             '          "mean" : '
                                             '16847.674926,\n'
                                             '          "stddev" : '
                                             '6639.843683,\n'
                                             '          "N" : 2121912\n'
                                             '        },\n'
                                             '        "bw_min" : 150945,\n'
                                             '        "bw_max" : 152439,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '151590.285714,\n'
                                             '        "bw_dev" : 485.280320,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 301890,\n'
                                             '        "iops_max" : 304878,\n'
                                             '        "iops_mean" : '
                                             '303180.571429,\n'
                                             '        "iops_stddev" : '
                                             '970.850675,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 40.557143,\n'
                                             '      "sys_cpu" : 59.414286,\n'
                                             '      "ctx" : 26,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.471274,\n'
                                             '        "2" : 26.285576,\n'
                                             '        "4" : 58.295997,\n'
                                             '        "8" : 14.947154,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 87.443970,\n'
                                             '        "8" : 12.556030,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.033743,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 6.563524,\n'
                                             '        "10" : 29.667955,\n'
                                             '        "20" : 42.265856,\n'
                                             '        "50" : 21.394131,\n'
                                             '        "100" : 0.072718,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=144056: Wed '
                                             'Oct 26 04:51:40 2022\n'
                                             '  read: IOPS=303k, BW=148MiB/s '
                                             '(155MB/s)(1036MiB/7001msec)\n'
                                             '    slat (nsec): min=1593, '
                                             'max=35116, avg=2769.19, '
                                             'stdev=1002.28\n'
                                             '    clat (nsec): min=330, '
                                             'max=188353, avg=14078.49, '
                                             'stdev=6683.08\n'
                                             '     lat (usec): min=6, max=191, '
                                             'avg=16.85, stdev= 6.64\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3408],  '
                                             '5.00th=[ 3536], 10.00th=[ 4832], '
                                             '20.00th=[ 6816],\n'
                                             '     | 30.00th=[ 9664], '
                                             '40.00th=[12096], '
                                             '50.00th=[14272], '
                                             '60.00th=[16320],\n'
                                             '     | 70.00th=[18816], '
                                             '80.00th=[20608], '
                                             '90.00th=[22912], '
                                             '95.00th=[23424],\n'
                                             '     | 99.00th=[27264], '
                                             '99.50th=[29824], '
                                             '99.90th=[42752], '
                                             '99.95th=[57088],\n'
                                             '     | 99.99th=[70144]\n'
                                             '   bw (  KiB/s): min=150945, '
                                             'max=152439, per=100.00%, '
                                             'avg=151590.29, stdev=485.28, '
                                             'samples=14\n'
                                             '   iops        : min=301890, '
                                             'max=304878, avg=303180.57, '
                                             'stdev=970.85, samples=14\n'
                                             '  lat (nsec)   : 500=0.03%, '
                                             '750=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=6.56%, 10=29.67%, 20=42.27%, '
                                             '50=21.39%\n'
                                             '  lat (usec)   : 100=0.07%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=40.56%, '
                                             'sys=59.41%, ctx=26, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.5%, '
                                             '2=26.3%, 4=58.3%, 8=14.9%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=87.4%, 8=12.6%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2121909,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=148MiB/s (155MB/s), '
                                             '148MiB/s-148MiB/s '
                                             '(155MB/s-155MB/s), io=1036MiB '
                                             '(1086MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.output'),
                                   'state': {'begin': 1666745610.9466832,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7992534637451172,
                                             'end': 1666745611.7459366,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_372.state')},
                       'cmd_373': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.output'),
                                   'state': {'begin': 1666745613.8330824,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7890000343322754,
                                             'end': 1666745614.6220825,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_373.state')},
                       'cmd_374': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.output'),
                                   'state': {'begin': 1666745614.6243002,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_aio_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.417046070098877,
                                             'end': 1666745626.0413463,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_374.state')},
                       'cmd_375': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752715,\n'
                                             '  "timestamp_ms" : '
                                             '1666752715116,\n'
                                             '  "time" : "Wed Oct 26 04:51:55 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1076485120,\n'
                                             '        "io_kbytes" : 1051255,\n'
                                             '        "bw_bytes" : 153761622,\n'
                                             '        "bw" : 150157,\n'
                                             '        "iops" : 300314.669333,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2102503,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1593,\n'
                                             '          "max" : 36158,\n'
                                             '          "mean" : 2832.892318,\n'
                                             '          "stddev" : '
                                             '1021.358270,\n'
                                             '          "N" : 2102503\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 350,\n'
                                             '          "max" : 128501,\n'
                                             '          "mean" : '
                                             '14078.870878,\n'
                                             '          "stddev" : '
                                             '6722.192899,\n'
                                             '          "N" : 2102503,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3440,\n'
                                             '            "5.000000" : 3568,\n'
                                             '            "10.000000" : 4832,\n'
                                             '            "20.000000" : 6816,\n'
                                             '            "30.000000" : 9664,\n'
                                             '            "40.000000" : '
                                             '11712,\n'
                                             '            "50.000000" : '
                                             '14144,\n'
                                             '            "60.000000" : '
                                             '16320,\n'
                                             '            "70.000000" : '
                                             '18304,\n'
                                             '            "80.000000" : '
                                             '19840,\n'
                                             '            "90.000000" : '
                                             '23168,\n'
                                             '            "95.000000" : '
                                             '23680,\n'
                                             '            "99.000000" : '
                                             '27008,\n'
                                             '            "99.500000" : '
                                             '29824,\n'
                                             '            "99.900000" : '
                                             '44800,\n'
                                             '            "99.950000" : '
                                             '57600,\n'
                                             '            "99.990000" : 69120\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6161,\n'
                                             '          "max" : 130174,\n'
                                             '          "mean" : '
                                             '16911.763196,\n'
                                             '          "stddev" : '
                                             '6736.191451,\n'
                                             '          "N" : 2102503\n'
                                             '        },\n'
                                             '        "bw_min" : 149226,\n'
                                             '        "bw_max" : 150989,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '150194.785714,\n'
                                             '        "bw_dev" : 508.754464,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 298452,\n'
                                             '        "iops_max" : 301979,\n'
                                             '        "iops_mean" : '
                                             '300389.642857,\n'
                                             '        "iops_stddev" : '
                                             '1017.629040,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 40.442857,\n'
                                             '      "sys_cpu" : 59.528571,\n'
                                             '      "ctx" : 29,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.466064,\n'
                                             '        "2" : 26.752542,\n'
                                             '        "4" : 57.926671,\n'
                                             '        "8" : 14.854723,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 87.516984,\n'
                                             '        "8" : 12.483016,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.042806,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 6.502393,\n'
                                             '        "10" : 29.740600,\n'
                                             '        "20" : 43.938059,\n'
                                             '        "50" : 19.695097,\n'
                                             '        "100" : 0.078906,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=144105: Wed '
                                             'Oct 26 04:51:55 2022\n'
                                             '  read: IOPS=300k, BW=147MiB/s '
                                             '(154MB/s)(1027MiB/7001msec)\n'
                                             '    slat (nsec): min=1593, '
                                             'max=36158, avg=2832.89, '
                                             'stdev=1021.36\n'
                                             '    clat (nsec): min=350, '
                                             'max=128501, avg=14078.87, '
                                             'stdev=6722.19\n'
                                             '     lat (usec): min=6, max=130, '
                                             'avg=16.91, stdev= 6.74\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3440],  '
                                             '5.00th=[ 3568], 10.00th=[ 4832], '
                                             '20.00th=[ 6816],\n'
                                             '     | 30.00th=[ 9664], '
                                             '40.00th=[11712], '
                                             '50.00th=[14144], '
                                             '60.00th=[16320],\n'
                                             '     | 70.00th=[18304], '
                                             '80.00th=[19840], '
                                             '90.00th=[23168], '
                                             '95.00th=[23680],\n'
                                             '     | 99.00th=[27008], '
                                             '99.50th=[29824], '
                                             '99.90th=[44800], '
                                             '99.95th=[57600],\n'
                                             '     | 99.99th=[69120]\n'
                                             '   bw (  KiB/s): min=149226, '
                                             'max=150989, per=100.00%, '
                                             'avg=150194.79, stdev=508.75, '
                                             'samples=14\n'
                                             '   iops        : min=298452, '
                                             'max=301979, avg=300389.64, '
                                             'stdev=1017.63, samples=14\n'
                                             '  lat (nsec)   : 500=0.04%, '
                                             '750=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=6.50%, 10=29.74%, 20=43.94%, '
                                             '50=19.70%\n'
                                             '  lat (usec)   : 100=0.08%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=40.44%, '
                                             'sys=59.53%, ctx=29, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.5%, '
                                             '2=26.8%, 4=57.9%, 8=14.9%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=87.5%, 8=12.5%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2102503,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=147MiB/s (154MB/s), '
                                             '147MiB/s-147MiB/s '
                                             '(154MB/s-154MB/s), io=1027MiB '
                                             '(1076MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.output'),
                                   'state': {'begin': 1666745626.044057,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7855679988861084,
                                             'end': 1666745626.829625,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_375.state')},
                       'cmd_376': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.output'),
                                   'state': {'begin': 1666745628.8473525,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7852327823638916,
                                             'end': 1666745629.6325853,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_376.state')},
                       'cmd_377': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.output'),
                                   'state': {'begin': 1666745629.6349015,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_aio_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.413702487945557,
                                             'end': 1666745641.048604,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_377.state')},
                       'cmd_378': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752730,\n'
                                             '  "timestamp_ms" : '
                                             '1666752730123,\n'
                                             '  "time" : "Wed Oct 26 04:52:10 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1091307008,\n'
                                             '        "io_kbytes" : 1065729,\n'
                                             '        "bw_bytes" : 155878732,\n'
                                             '        "bw" : 152225,\n'
                                             '        "iops" : 304450.078560,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2131455,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1572,\n'
                                             '          "max" : 80341,\n'
                                             '          "mean" : 2723.375660,\n'
                                             '          "stddev" : '
                                             '926.510920,\n'
                                             '          "N" : 2131456\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 320,\n'
                                             '          "max" : 145703,\n'
                                             '          "mean" : '
                                             '14031.642536,\n'
                                             '          "stddev" : '
                                             '6669.051166,\n'
                                             '          "N" : 2131459,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3376,\n'
                                             '            "5.000000" : 3504,\n'
                                             '            "10.000000" : 4832,\n'
                                             '            "20.000000" : 6688,\n'
                                             '            "30.000000" : 9536,\n'
                                             '            "40.000000" : '
                                             '11968,\n'
                                             '            "50.000000" : '
                                             '14144,\n'
                                             '            "60.000000" : '
                                             '16512,\n'
                                             '            "70.000000" : '
                                             '18560,\n'
                                             '            "80.000000" : '
                                             '20352,\n'
                                             '            "90.000000" : '
                                             '22912,\n'
                                             '            "95.000000" : '
                                             '23424,\n'
                                             '            "99.000000" : '
                                             '27008,\n'
                                             '            "99.500000" : '
                                             '29312,\n'
                                             '            "99.900000" : '
                                             '45312,\n'
                                             '            "99.950000" : '
                                             '58624,\n'
                                             '            "99.990000" : 70144\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6122,\n'
                                             '          "max" : 147336,\n'
                                             '          "mean" : '
                                             '16755.017869,\n'
                                             '          "stddev" : '
                                             '6618.416459,\n'
                                             '          "N" : 2131459\n'
                                             '        },\n'
                                             '        "bw_min" : 151543,\n'
                                             '        "bw_max" : 152997,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '152264.642857,\n'
                                             '        "bw_dev" : 377.366300,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 303086,\n'
                                             '        "iops_max" : 305994,\n'
                                             '        "iops_mean" : '
                                             '304529.357143,\n'
                                             '        "iops_stddev" : '
                                             '754.613463,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 41.871429,\n'
                                             '      "sys_cpu" : 58.100000,\n'
                                             '      "ctx" : 26,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.501066,\n'
                                             '        "2" : 26.540462,\n'
                                             '        "4" : 58.078777,\n'
                                             '        "8" : 14.879695,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 87.483879,\n'
                                             '        "8" : 12.516121,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.044148,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 6.314982,\n'
                                             '        "10" : 29.727393,\n'
                                             '        "20" : 43.166475,\n'
                                             '        "50" : 20.663397,\n'
                                             '        "100" : 0.081963,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=144155: Wed '
                                             'Oct 26 04:52:10 2022\n'
                                             '  read: IOPS=304k, BW=149MiB/s '
                                             '(156MB/s)(1041MiB/7001msec)\n'
                                             '    slat (nsec): min=1572, '
                                             'max=80341, avg=2723.38, '
                                             'stdev=926.51\n'
                                             '    clat (nsec): min=320, '
                                             'max=145703, avg=14031.64, '
                                             'stdev=6669.05\n'
                                             '     lat (usec): min=6, max=147, '
                                             'avg=16.76, stdev= 6.62\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3376],  '
                                             '5.00th=[ 3504], 10.00th=[ 4832], '
                                             '20.00th=[ 6688],\n'
                                             '     | 30.00th=[ 9536], '
                                             '40.00th=[11968], '
                                             '50.00th=[14144], '
                                             '60.00th=[16512],\n'
                                             '     | 70.00th=[18560], '
                                             '80.00th=[20352], '
                                             '90.00th=[22912], '
                                             '95.00th=[23424],\n'
                                             '     | 99.00th=[27008], '
                                             '99.50th=[29312], '
                                             '99.90th=[45312], '
                                             '99.95th=[58624],\n'
                                             '     | 99.99th=[70144]\n'
                                             '   bw (  KiB/s): min=151543, '
                                             'max=152997, per=100.00%, '
                                             'avg=152264.64, stdev=377.37, '
                                             'samples=14\n'
                                             '   iops        : min=303086, '
                                             'max=305994, avg=304529.36, '
                                             'stdev=754.61, samples=14\n'
                                             '  lat (nsec)   : 500=0.04%, '
                                             '750=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=6.31%, 10=29.73%, 20=43.17%, '
                                             '50=20.66%\n'
                                             '  lat (usec)   : 100=0.08%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=41.87%, '
                                             'sys=58.10%, ctx=26, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.5%, '
                                             '2=26.5%, 4=58.1%, 8=14.9%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=87.5%, 8=12.5%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2131455,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=149MiB/s (156MB/s), '
                                             '149MiB/s-149MiB/s '
                                             '(156MB/s-156MB/s), io=1041MiB '
                                             '(1091MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.output'),
                                   'state': {'begin': 1666745641.0516608,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8644020557403564,
                                             'end': 1666745641.9160628,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_378.state')},
                       'cmd_379': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.output'),
                                   'state': {'begin': 1666745644.222322,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8765056133270264,
                                             'end': 1666745645.0988276,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_379.state')},
                       'cmd_38': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.output'),
                                  'state': {'begin': 1666744008.8447871,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_uring" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.455432415008545,
                                            'end': 1666744020.3002195,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_38.state')},
                       'cmd_380': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.output'),
                                   'state': {'begin': 1666745645.1010451,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.647058725357056,
                                             'end': 1666745656.7481039,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_380.state')},
                       'cmd_381': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752745,\n'
                                             '  "timestamp_ms" : '
                                             '1666752745825,\n'
                                             '  "time" : "Wed Oct 26 04:52:25 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1052019200,\n'
                                             '        "io_kbytes" : 1027362,\n'
                                             '        "bw_bytes" : 150266990,\n'
                                             '        "bw" : 146745,\n'
                                             '        "iops" : 293489.072990,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2054717,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1623,\n'
                                             '          "max" : 44804,\n'
                                             '          "mean" : 2930.171240,\n'
                                             '          "stddev" : '
                                             '1093.681472,\n'
                                             '          "N" : 2054718\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 751,\n'
                                             '          "max" : 124714,\n'
                                             '          "mean" : '
                                             '14597.804246,\n'
                                             '          "stddev" : '
                                             '6757.196878,\n'
                                             '          "N" : 2054717,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3920,\n'
                                             '            "5.000000" : 4048,\n'
                                             '            "10.000000" : 5280,\n'
                                             '            "20.000000" : 8256,\n'
                                             '            "30.000000" : '
                                             '10176,\n'
                                             '            "40.000000" : '
                                             '12096,\n'
                                             '            "50.000000" : '
                                             '14656,\n'
                                             '            "60.000000" : '
                                             '16768,\n'
                                             '            "70.000000" : '
                                             '18560,\n'
                                             '            "80.000000" : '
                                             '20096,\n'
                                             '            "90.000000" : '
                                             '24192,\n'
                                             '            "95.000000" : '
                                             '24448,\n'
                                             '            "99.000000" : '
                                             '27264,\n'
                                             '            "99.500000" : '
                                             '30336,\n'
                                             '            "99.900000" : '
                                             '42752,\n'
                                             '            "99.950000" : '
                                             '56576,\n'
                                             '            "99.990000" : 69120\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6472,\n'
                                             '          "max" : 126397,\n'
                                             '          "mean" : '
                                             '17527.975420,\n'
                                             '          "stddev" : '
                                             '6799.693481,\n'
                                             '          "N" : 2054717\n'
                                             '        },\n'
                                             '        "bw_min" : 146440,\n'
                                             '        "bw_max" : 147365,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '146789.214286,\n'
                                             '        "bw_dev" : 284.407935,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 292880,\n'
                                             '        "iops_max" : 294730,\n'
                                             '        "iops_mean" : '
                                             '293578.500000,\n'
                                             '        "iops_stddev" : '
                                             '568.809925,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.014286,\n'
                                             '      "sys_cpu" : 60.957143,\n'
                                             '      "ctx" : 27,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.407453,\n'
                                             '        "2" : 26.810310,\n'
                                             '        "4" : 57.959417,\n'
                                             '        "8" : 14.822820,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 87.512790,\n'
                                             '        "8" : 12.487210,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.029931\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 3.778185,\n'
                                             '        "10" : 23.005261,\n'
                                             '        "20" : 50.434439,\n'
                                             '        "50" : 22.678257,\n'
                                             '        "100" : 0.072613,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=144204: Wed '
                                             'Oct 26 04:52:25 2022\n'
                                             '  read: IOPS=293k, BW=143MiB/s '
                                             '(150MB/s)(1003MiB/7001msec)\n'
                                             '    slat (nsec): min=1623, '
                                             'max=44804, avg=2930.17, '
                                             'stdev=1093.68\n'
                                             '    clat (nsec): min=751, '
                                             'max=124714, avg=14597.80, '
                                             'stdev=6757.20\n'
                                             '     lat (usec): min=6, max=126, '
                                             'avg=17.53, stdev= 6.80\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3920],  '
                                             '5.00th=[ 4048], 10.00th=[ 5280], '
                                             '20.00th=[ 8256],\n'
                                             '     | 30.00th=[10176], '
                                             '40.00th=[12096], '
                                             '50.00th=[14656], '
                                             '60.00th=[16768],\n'
                                             '     | 70.00th=[18560], '
                                             '80.00th=[20096], '
                                             '90.00th=[24192], '
                                             '95.00th=[24448],\n'
                                             '     | 99.00th=[27264], '
                                             '99.50th=[30336], '
                                             '99.90th=[42752], '
                                             '99.95th=[56576],\n'
                                             '     | 99.99th=[69120]\n'
                                             '   bw (  KiB/s): min=146440, '
                                             'max=147365, per=100.00%, '
                                             'avg=146789.21, stdev=284.41, '
                                             'samples=14\n'
                                             '   iops        : min=292880, '
                                             'max=294730, avg=293578.50, '
                                             'stdev=568.81, samples=14\n'
                                             '  lat (nsec)   : 1000=0.03%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=3.78%, 10=23.01%, 20=50.43%, '
                                             '50=22.68%\n'
                                             '  lat (usec)   : 100=0.07%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=39.01%, '
                                             'sys=60.96%, ctx=27, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.4%, '
                                             '2=26.8%, 4=58.0%, 8=14.8%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=87.5%, 8=12.5%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2054717,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=143MiB/s (150MB/s), '
                                             '143MiB/s-143MiB/s '
                                             '(150MB/s-150MB/s), io=1003MiB '
                                             '(1052MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.output'),
                                   'state': {'begin': 1666745656.7511828,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8129782676696777,
                                             'end': 1666745657.564161,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_381.state')},
                       'cmd_382': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.output'),
                                   'state': {'begin': 1666745659.6146078,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7873172760009766,
                                             'end': 1666745660.401925,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_382.state')},
                       'cmd_383': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.output'),
                                   'state': {'begin': 1666745660.4042623,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.560972213745117,
                                             'end': 1666745671.9652345,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_383.state')},
                       'cmd_384': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752761,\n'
                                             '  "timestamp_ms" : '
                                             '1666752761025,\n'
                                             '  "time" : "Wed Oct 26 04:52:41 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1073317888,\n'
                                             '        "io_kbytes" : 1048162,\n'
                                             '        "bw_bytes" : 153309225,\n'
                                             '        "bw" : 149716,\n'
                                             '        "iops" : 299431.366948,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2096319,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1593,\n'
                                             '          "max" : 80861,\n'
                                             '          "mean" : 2771.613320,\n'
                                             '          "stddev" : '
                                             '938.858044,\n'
                                             '          "N" : 2096320\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 732,\n'
                                             '          "max" : 127409,\n'
                                             '          "mean" : '
                                             '14376.397846,\n'
                                             '          "stddev" : '
                                             '6697.479279,\n'
                                             '          "N" : 2096324,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3792,\n'
                                             '            "5.000000" : 3888,\n'
                                             '            "10.000000" : 5152,\n'
                                             '            "20.000000" : 7200,\n'
                                             '            "30.000000" : '
                                             '10048,\n'
                                             '            "40.000000" : '
                                             '11840,\n'
                                             '            "50.000000" : '
                                             '14528,\n'
                                             '            "60.000000" : '
                                             '16768,\n'
                                             '            "70.000000" : '
                                             '18304,\n'
                                             '            "80.000000" : '
                                             '19840,\n'
                                             '            "90.000000" : '
                                             '23680,\n'
                                             '            "95.000000" : '
                                             '24192,\n'
                                             '            "99.000000" : '
                                             '27264,\n'
                                             '            "99.500000" : '
                                             '29824,\n'
                                             '            "99.900000" : '
                                             '41728,\n'
                                             '            "99.950000" : '
                                             '56576,\n'
                                             '            "99.990000" : 70144\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6502,\n'
                                             '          "max" : 129052,\n'
                                             '          "mean" : '
                                             '17148.011537,\n'
                                             '          "stddev" : '
                                             '6676.946354,\n'
                                             '          "N" : 2096324\n'
                                             '        },\n'
                                             '        "bw_min" : 149307,\n'
                                             '        "bw_max" : 150245,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '149750.857143,\n'
                                             '        "bw_dev" : 298.817891,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 298614,\n'
                                             '        "iops_max" : 300490,\n'
                                             '        "iops_mean" : '
                                             '299501.714286,\n'
                                             '        "iops_stddev" : '
                                             '597.243338,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.985714,\n'
                                             '      "sys_cpu" : 59.985714,\n'
                                             '      "ctx" : 25,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.448739,\n'
                                             '        "2" : 26.840428,\n'
                                             '        "4" : 57.885036,\n'
                                             '        "8" : 14.825797,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 87.522870,\n'
                                             '        "8" : 12.477130,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.039975\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 6.707996,\n'
                                             '        "10" : 23.033327,\n'
                                             '        "20" : 51.503469,\n'
                                             '        "50" : 18.641581,\n'
                                             '        "100" : 0.071936,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=144253: Wed '
                                             'Oct 26 04:52:41 2022\n'
                                             '  read: IOPS=299k, BW=146MiB/s '
                                             '(153MB/s)(1024MiB/7001msec)\n'
                                             '    slat (nsec): min=1593, '
                                             'max=80861, avg=2771.61, '
                                             'stdev=938.86\n'
                                             '    clat (nsec): min=732, '
                                             'max=127409, avg=14376.40, '
                                             'stdev=6697.48\n'
                                             '     lat (usec): min=6, max=129, '
                                             'avg=17.15, stdev= 6.68\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3792],  '
                                             '5.00th=[ 3888], 10.00th=[ 5152], '
                                             '20.00th=[ 7200],\n'
                                             '     | 30.00th=[10048], '
                                             '40.00th=[11840], '
                                             '50.00th=[14528], '
                                             '60.00th=[16768],\n'
                                             '     | 70.00th=[18304], '
                                             '80.00th=[19840], '
                                             '90.00th=[23680], '
                                             '95.00th=[24192],\n'
                                             '     | 99.00th=[27264], '
                                             '99.50th=[29824], '
                                             '99.90th=[41728], '
                                             '99.95th=[56576],\n'
                                             '     | 99.99th=[70144]\n'
                                             '   bw (  KiB/s): min=149307, '
                                             'max=150245, per=100.00%, '
                                             'avg=149750.86, stdev=298.82, '
                                             'samples=14\n'
                                             '   iops        : min=298614, '
                                             'max=300490, avg=299501.71, '
                                             'stdev=597.24, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.04%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=6.71%, 10=23.03%, 20=51.50%, '
                                             '50=18.64%\n'
                                             '  lat (usec)   : 100=0.07%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=39.99%, '
                                             'sys=59.99%, ctx=25, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.4%, '
                                             '2=26.8%, 4=57.9%, 8=14.8%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=87.5%, 8=12.5%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2096319,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=146MiB/s (153MB/s), '
                                             '146MiB/s-146MiB/s '
                                             '(153MB/s-153MB/s), io=1024MiB '
                                             '(1073MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.output'),
                                   'state': {'begin': 1666745671.9682,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7915475368499756,
                                             'end': 1666745672.7597475,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_384.state')},
                       'cmd_385': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.output'),
                                   'state': {'begin': 1666745674.8022122,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7958424091339111,
                                             'end': 1666745675.5980546,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_385.state')},
                       'cmd_386': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.output'),
                                   'state': {'begin': 1666745675.6002083,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.533339262008667,
                                             'end': 1666745687.1335475,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_386.state')},
                       'cmd_387': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=spdk_bdev, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752776,\n'
                                             '  "timestamp_ms" : '
                                             '1666752776202,\n'
                                             '  "time" : "Wed Oct 26 04:52:56 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1060783104,\n'
                                             '        "io_kbytes" : 1035921,\n'
                                             '        "bw_bytes" : 151518797,\n'
                                             '        "bw" : 147967,\n'
                                             '        "iops" : 295934.152264,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 2071835,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1623,\n'
                                             '          "max" : 49042,\n'
                                             '          "mean" : 2860.583254,\n'
                                             '          "stddev" : '
                                             '1041.203387,\n'
                                             '          "N" : 2071836\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 762,\n'
                                             '          "max" : 127149,\n'
                                             '          "mean" : '
                                             '14498.881318,\n'
                                             '          "stddev" : '
                                             '6774.396456,\n'
                                             '          "N" : 2071842,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 3856,\n'
                                             '            "5.000000" : 3984,\n'
                                             '            "10.000000" : 5216,\n'
                                             '            "20.000000" : 8256,\n'
                                             '            "30.000000" : '
                                             '10176,\n'
                                             '            "40.000000" : '
                                             '11968,\n'
                                             '            "50.000000" : '
                                             '14528,\n'
                                             '            "60.000000" : '
                                             '17280,\n'
                                             '            "70.000000" : '
                                             '18560,\n'
                                             '            "80.000000" : '
                                             '19840,\n'
                                             '            "90.000000" : '
                                             '23936,\n'
                                             '            "95.000000" : '
                                             '24448,\n'
                                             '            "99.000000" : '
                                             '27520,\n'
                                             '            "99.500000" : '
                                             '30592,\n'
                                             '            "99.900000" : '
                                             '46336,\n'
                                             '            "99.950000" : '
                                             '58624,\n'
                                             '            "99.990000" : 72192\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6472,\n'
                                             '          "max" : 128822,\n'
                                             '          "mean" : '
                                             '17359.464180,\n'
                                             '          "stddev" : '
                                             '6757.687788,\n'
                                             '          "N" : 2071842\n'
                                             '        },\n'
                                             '        "bw_min" : 147584,\n'
                                             '        "bw_max" : 148654,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '148008.500000,\n'
                                             '        "bw_dev" : 324.912710,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 295168,\n'
                                             '        "iops_max" : 297308,\n'
                                             '        "iops_mean" : '
                                             '296017.000000,\n'
                                             '        "iops_stddev" : '
                                             '650.166251,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.242857,\n'
                                             '      "sys_cpu" : 60.728571,\n'
                                             '      "ctx" : 24,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.416249,\n'
                                             '        "2" : 26.862419,\n'
                                             '        "4" : 57.902777,\n'
                                             '        "8" : 14.818555,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 87.526588,\n'
                                             '        "8" : 12.473412,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.035331\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 5.719471,\n'
                                             '        "10" : 21.807528,\n'
                                             '        "20" : 52.966718,\n'
                                             '        "50" : 19.386679,\n'
                                             '        "100" : 0.083260,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=144302: Wed '
                                             'Oct 26 04:52:56 2022\n'
                                             '  read: IOPS=296k, BW=144MiB/s '
                                             '(152MB/s)(1012MiB/7001msec)\n'
                                             '    slat (nsec): min=1623, '
                                             'max=49042, avg=2860.58, '
                                             'stdev=1041.20\n'
                                             '    clat (nsec): min=762, '
                                             'max=127149, avg=14498.88, '
                                             'stdev=6774.40\n'
                                             '     lat (usec): min=6, max=128, '
                                             'avg=17.36, stdev= 6.76\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[ 3856],  '
                                             '5.00th=[ 3984], 10.00th=[ 5216], '
                                             '20.00th=[ 8256],\n'
                                             '     | 30.00th=[10176], '
                                             '40.00th=[11968], '
                                             '50.00th=[14528], '
                                             '60.00th=[17280],\n'
                                             '     | 70.00th=[18560], '
                                             '80.00th=[19840], '
                                             '90.00th=[23936], '
                                             '95.00th=[24448],\n'
                                             '     | 99.00th=[27520], '
                                             '99.50th=[30592], '
                                             '99.90th=[46336], '
                                             '99.95th=[58624],\n'
                                             '     | 99.99th=[72192]\n'
                                             '   bw (  KiB/s): min=147584, '
                                             'max=148654, per=100.00%, '
                                             'avg=148008.50, stdev=324.91, '
                                             'samples=14\n'
                                             '   iops        : min=295168, '
                                             'max=297308, avg=296017.00, '
                                             'stdev=650.17, samples=14\n'
                                             '  lat (nsec)   : 1000=0.04%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=5.72%, 10=21.81%, 20=52.97%, '
                                             '50=19.39%\n'
                                             '  lat (usec)   : 100=0.08%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=39.24%, '
                                             'sys=60.73%, ctx=24, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.4%, '
                                             '2=26.9%, 4=57.9%, 8=14.8%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=87.5%, 8=12.5%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=2071835,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=144MiB/s (152MB/s), '
                                             '144MiB/s-144MiB/s '
                                             '(152MB/s-152MB/s), io=1012MiB '
                                             '(1061MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.output'),
                                   'state': {'begin': 1666745687.1367385,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7910754680633545,
                                             'end': 1666745687.927814,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_387.state')},
                       'cmd_388': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.output'),
                                   'state': {'begin': 1666745688.926372,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7844598293304443,
                                             'end': 1666745689.7108319,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_388.state')},
                       'cmd_389': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.output'),
                                   'state': {'begin': 1666745689.7130206,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-xnvme_0.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.091380596160889,
                                             'end': 1666745700.8044012,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_389.state')},
                       'cmd_39': {'output': 'io_uring-bdev_uring: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751109,\n'
                                            '  "timestamp_ms" : '
                                            '1666751109430,\n'
                                            '  "time" : "Wed Oct 26 04:25:09 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_uring",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_uring",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 554295808,\n'
                                            '        "io_kbytes" : 541304,\n'
                                            '        "bw_bytes" : 79173804,\n'
                                            '        "bw" : 77318,\n'
                                            '        "iops" : 154636.337666,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1082609,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 160,\n'
                                            '          "max" : 11602,\n'
                                            '          "mean" : 177.716369,\n'
                                            '          "stddev" : 44.536553,\n'
                                            '          "N" : 1082609\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 5681,\n'
                                            '          "max" : 127730,\n'
                                            '          "mean" : 6024.580633,\n'
                                            '          "stddev" : '
                                            '1112.814196,\n'
                                            '          "N" : 1082609,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 5728,\n'
                                            '            "5.000000" : 5792,\n'
                                            '            "10.000000" : 5792,\n'
                                            '            "20.000000" : 5792,\n'
                                            '            "30.000000" : 5792,\n'
                                            '            "40.000000" : 5856,\n'
                                            '            "50.000000" : 5856,\n'
                                            '            "60.000000" : 5920,\n'
                                            '            "70.000000" : 5920,\n'
                                            '            "80.000000" : 5920,\n'
                                            '            "90.000000" : 6048,\n'
                                            '            "95.000000" : 6112,\n'
                                            '            "99.000000" : 11328,\n'
                                            '            "99.500000" : 12608,\n'
                                            '            "99.900000" : 17024,\n'
                                            '            "99.950000" : 17024,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5851,\n'
                                            '          "max" : 127900,\n'
                                            '          "mean" : 6202.297002,\n'
                                            '          "stddev" : '
                                            '1114.031677,\n'
                                            '          "N" : 1082609\n'
                                            '        },\n'
                                            '        "bw_min" : 77106,\n'
                                            '        "bw_max" : 77614,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '77336.642857,\n'
                                            '        "bw_dev" : 146.503141,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 154212,\n'
                                            '        "iops_max" : 155228,\n'
                                            '        "iops_mean" : '
                                            '154673.285714,\n'
                                            '        "iops_stddev" : '
                                            '293.006282,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 51.814286,\n'
                                            '      "sys_cpu" : 48.157143,\n'
                                            '      "ctx" : 21,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.603836,\n'
                                            '        "20" : 1.381108,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.012378,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_uring: (groupid=0, '
                                            'jobs=1): err= 0: pid=139184: Wed '
                                            'Oct 26 04:25:09 2022\n'
                                            '  read: IOPS=155k, BW=75.5MiB/s '
                                            '(79.2MB/s)(529MiB/7001msec)\n'
                                            '    slat (nsec): min=160, '
                                            'max=11602, avg=177.72, '
                                            'stdev=44.54\n'
                                            '    clat (usec): min=5, max=127, '
                                            'avg= 6.02, stdev= 1.11\n'
                                            '     lat (usec): min=5, max=127, '
                                            'avg= 6.20, stdev= 1.11\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 5728],  5.00th=[ '
                                            '5792], 10.00th=[ 5792], 20.00th=[ '
                                            '5792],\n'
                                            '     | 30.00th=[ 5792], 40.00th=[ '
                                            '5856], 50.00th=[ 5856], 60.00th=[ '
                                            '5920],\n'
                                            '     | 70.00th=[ 5920], 80.00th=[ '
                                            '5920], 90.00th=[ 6048], 95.00th=[ '
                                            '6112],\n'
                                            '     | 99.00th=[11328], '
                                            '99.50th=[12608], 99.90th=[17024], '
                                            '99.95th=[17024],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=77106, '
                                            'max=77614, per=100.00%, '
                                            'avg=77336.64, stdev=146.50, '
                                            'samples=14\n'
                                            '   iops        : min=154212, '
                                            'max=155228, avg=154673.29, '
                                            'stdev=293.01, samples=14\n'
                                            '  lat (usec)   : 10=98.60%, '
                                            '20=1.38%, 50=0.01%, 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=51.81%, '
                                            'sys=48.16%, ctx=21, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1082609,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=75.5MiB/s (79.2MB/s), '
                                            '75.5MiB/s-75.5MiB/s '
                                            '(79.2MB/s-79.2MB/s), io=529MiB '
                                            '(554MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.output'),
                                  'state': {'begin': 1666744020.303161,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8060753345489502,
                                            'end': 1666744021.1092362,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_39.state')},
                       'cmd_390': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752789,\n'
                                             '  "timestamp_ms" : '
                                             '1666752789979,\n'
                                             '  "time" : "Wed Oct 26 04:53:09 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1014642688,\n'
                                             '        "io_kbytes" : 990862,\n'
                                             '        "bw_bytes" : 144928251,\n'
                                             '        "bw" : 141531,\n'
                                             '        "iops" : 283061.848307,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1981716,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1472,\n'
                                             '          "max" : 32762,\n'
                                             '          "mean" : 2358.838814,\n'
                                             '          "stddev" : '
                                             '1033.707278,\n'
                                             '          "N" : 1981717\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 6823,\n'
                                             '          "max" : 153127,\n'
                                             '          "mean" : '
                                             '25221.540123,\n'
                                             '          "stddev" : '
                                             '3166.343719,\n'
                                             '          "N" : 1981723,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 18304,\n'
                                             '            "5.000000" : 23168,\n'
                                             '            "10.000000" : '
                                             '23424,\n'
                                             '            "20.000000" : '
                                             '23424,\n'
                                             '            "30.000000" : '
                                             '23680,\n'
                                             '            "40.000000" : '
                                             '23680,\n'
                                             '            "50.000000" : '
                                             '24960,\n'
                                             '            "60.000000" : '
                                             '25984,\n'
                                             '            "70.000000" : '
                                             '26240,\n'
                                             '            "80.000000" : '
                                             '26240,\n'
                                             '            "90.000000" : '
                                             '26496,\n'
                                             '            "95.000000" : '
                                             '29568,\n'
                                             '            "99.000000" : '
                                             '38144,\n'
                                             '            "99.500000" : '
                                             '42752,\n'
                                             '            "99.900000" : '
                                             '51968,\n'
                                             '            "99.950000" : '
                                             '64768,\n'
                                             '            "99.990000" : 78336\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10910,\n'
                                             '          "max" : 154660,\n'
                                             '          "mean" : '
                                             '27580.380426,\n'
                                             '          "stddev" : '
                                             '2971.138538,\n'
                                             '          "N" : 1981723\n'
                                             '        },\n'
                                             '        "bw_min" : 141148,\n'
                                             '        "bw_max" : 141868,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '141569.500000,\n'
                                             '        "bw_dev" : 202.955755,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 282296,\n'
                                             '        "iops_max" : 283736,\n'
                                             '        "iops_mean" : '
                                             '283139.142857,\n'
                                             '        "iops_stddev" : '
                                             '405.859175,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 36.985714,\n'
                                             '      "sys_cpu" : 62.000000,\n'
                                             '      "ctx" : 9204,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999975,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 1.889827,\n'
                                             '        "50" : 97.980891,\n'
                                             '        "100" : 0.126507,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=144341: Wed '
                                             'Oct 26 04:53:09 2022\n'
                                             '  read: IOPS=283k, BW=138MiB/s '
                                             '(145MB/s)(968MiB/7001msec)\n'
                                             '    slat (nsec): min=1472, '
                                             'max=32762, avg=2358.84, '
                                             'stdev=1033.71\n'
                                             '    clat (usec): min=6, max=153, '
                                             'avg=25.22, stdev= 3.17\n'
                                             '     lat (usec): min=10, '
                                             'max=154, avg=27.58, stdev= 2.97\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[18304],  '
                                             '5.00th=[23168], 10.00th=[23424], '
                                             '20.00th=[23424],\n'
                                             '     | 30.00th=[23680], '
                                             '40.00th=[23680], '
                                             '50.00th=[24960], '
                                             '60.00th=[25984],\n'
                                             '     | 70.00th=[26240], '
                                             '80.00th=[26240], '
                                             '90.00th=[26496], '
                                             '95.00th=[29568],\n'
                                             '     | 99.00th=[38144], '
                                             '99.50th=[42752], '
                                             '99.90th=[51968], '
                                             '99.95th=[64768],\n'
                                             '     | 99.99th=[78336]\n'
                                             '   bw (  KiB/s): min=141148, '
                                             'max=141868, per=100.00%, '
                                             'avg=141569.50, stdev=202.96, '
                                             'samples=14\n'
                                             '   iops        : min=282296, '
                                             'max=283736, avg=283139.14, '
                                             'stdev=405.86, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=1.89%, 50=97.98%, 100=0.13%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=36.99%, '
                                             'sys=62.00%, ctx=9204, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1981716,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=138MiB/s (145MB/s), '
                                             '138MiB/s-138MiB/s '
                                             '(145MB/s-145MB/s), io=968MiB '
                                             '(1015MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.output'),
                                   'state': {'begin': 1666745700.8076355,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8106334209442139,
                                             'end': 1666745701.618269,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_390.state')},
                       'cmd_391': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.output'),
                                   'state': {'begin': 1666745702.6029127,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7850196361541748,
                                             'end': 1666745703.3879323,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_391.state')},
                       'cmd_392': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.output'),
                                   'state': {'begin': 1666745703.3901885,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-xnvme_1.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.085580110549927,
                                             'end': 1666745714.4757686,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_392.state')},
                       'cmd_393': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752803,\n'
                                             '  "timestamp_ms" : '
                                             '1666752803651,\n'
                                             '  "time" : "Wed Oct 26 04:53:23 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1012346368,\n'
                                             '        "io_kbytes" : 988619,\n'
                                             '        "bw_bytes" : 144600252,\n'
                                             '        "bw" : 141211,\n'
                                             '        "iops" : 282421.368376,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1977232,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1482,\n'
                                             '          "max" : 54362,\n'
                                             '          "mean" : 2372.692545,\n'
                                             '          "stddev" : '
                                             '1039.385184,\n'
                                             '          "N" : 1977232\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4679,\n'
                                             '          "max" : 146735,\n'
                                             '          "mean" : '
                                             '25274.840964,\n'
                                             '          "stddev" : '
                                             '3228.285563,\n'
                                             '          "N" : 1977239,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 18304,\n'
                                             '            "5.000000" : 23168,\n'
                                             '            "10.000000" : '
                                             '23424,\n'
                                             '            "20.000000" : '
                                             '23424,\n'
                                             '            "30.000000" : '
                                             '23680,\n'
                                             '            "40.000000" : '
                                             '23936,\n'
                                             '            "50.000000" : '
                                             '24960,\n'
                                             '            "60.000000" : '
                                             '25984,\n'
                                             '            "70.000000" : '
                                             '26240,\n'
                                             '            "80.000000" : '
                                             '26240,\n'
                                             '            "90.000000" : '
                                             '26496,\n'
                                             '            "95.000000" : '
                                             '30080,\n'
                                             '            "99.000000" : '
                                             '38656,\n'
                                             '            "99.500000" : '
                                             '42752,\n'
                                             '            "99.900000" : '
                                             '53504,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 6211,\n'
                                             '          "max" : 151143,\n'
                                             '          "mean" : '
                                             '27647.531768,\n'
                                             '          "stddev" : '
                                             '3048.297465,\n'
                                             '          "N" : 1977239\n'
                                             '        },\n'
                                             '        "bw_min" : 140956,\n'
                                             '        "bw_max" : 141621,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '141248.214286,\n'
                                             '        "bw_dev" : 159.472293,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 281912,\n'
                                             '        "iops_max" : 283242,\n'
                                             '        "iops_mean" : '
                                             '282496.500000,\n'
                                             '        "iops_stddev" : '
                                             '318.744351,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 38.000000,\n'
                                             '      "sys_cpu" : 60.957143,\n'
                                             '      "ctx" : 9034,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999975,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 1.892090,\n'
                                             '        "50" : 97.959976,\n'
                                             '        "100" : 0.144242,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=144380: Wed '
                                             'Oct 26 04:53:23 2022\n'
                                             '  read: IOPS=282k, BW=138MiB/s '
                                             '(145MB/s)(965MiB/7001msec)\n'
                                             '    slat (nsec): min=1482, '
                                             'max=54362, avg=2372.69, '
                                             'stdev=1039.39\n'
                                             '    clat (usec): min=4, max=146, '
                                             'avg=25.27, stdev= 3.23\n'
                                             '     lat (usec): min=6, max=151, '
                                             'avg=27.65, stdev= 3.05\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[18304],  '
                                             '5.00th=[23168], 10.00th=[23424], '
                                             '20.00th=[23424],\n'
                                             '     | 30.00th=[23680], '
                                             '40.00th=[23936], '
                                             '50.00th=[24960], '
                                             '60.00th=[25984],\n'
                                             '     | 70.00th=[26240], '
                                             '80.00th=[26240], '
                                             '90.00th=[26496], '
                                             '95.00th=[30080],\n'
                                             '     | 99.00th=[38656], '
                                             '99.50th=[42752], '
                                             '99.90th=[53504], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=140956, '
                                             'max=141621, per=100.00%, '
                                             'avg=141248.21, stdev=159.47, '
                                             'samples=14\n'
                                             '   iops        : min=281912, '
                                             'max=283242, avg=282496.50, '
                                             'stdev=318.74, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=1.89%, 50=97.96%, 100=0.14%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=38.00%, '
                                             'sys=60.96%, ctx=9034, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1977232,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=138MiB/s (145MB/s), '
                                             '138MiB/s-138MiB/s '
                                             '(145MB/s-145MB/s), io=965MiB '
                                             '(1012MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.output'),
                                   'state': {'begin': 1666745714.4786277,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7916066646575928,
                                             'end': 1666745715.2702343,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_393.state')},
                       'cmd_394': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.output'),
                                   'state': {'begin': 1666745716.243132,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7940242290496826,
                                             'end': 1666745717.0371563,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_394.state')},
                       'cmd_395': {'output': '',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.output'),
                                   'state': {'begin': 1666745717.0390606,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="512" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_512_8_libaio-xnvme_2.txt" '
                                                    '--cpus_allowed="1"',
                                             'cwd': 'None',
                                             'elapsed': 11.08297610282898,
                                             'end': 1666745728.1220367,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_395.state')},
                       'cmd_396': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 512B-512B, '
                                             '(W) 512B-512B, (T) 512B-512B, '
                                             'ioengine=xnvme, iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666752817,\n'
                                             '  "timestamp_ms" : '
                                             '1666752817296,\n'
                                             '  "time" : "Wed Oct 26 04:53:37 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "512",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1",\n'
                                             '        "cpus_allowed" : "1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '1011662336,\n'
                                             '        "io_kbytes" : 987951,\n'
                                             '        "bw_bytes" : 144502547,\n'
                                             '        "bw" : 141115,\n'
                                             '        "iops" : 282230.395658,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1975895,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1482,\n'
                                             '          "max" : 25537,\n'
                                             '          "mean" : 2298.559704,\n'
                                             '          "stddev" : '
                                             '931.379939,\n'
                                             '          "N" : 1975896\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4278,\n'
                                             '          "max" : 139502,\n'
                                             '          "mean" : '
                                             '25292.891398,\n'
                                             '          "stddev" : '
                                             '3159.053006,\n'
                                             '          "N" : 1975902,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 18304,\n'
                                             '            "5.000000" : 23168,\n'
                                             '            "10.000000" : '
                                             '23424,\n'
                                             '            "20.000000" : '
                                             '23680,\n'
                                             '            "30.000000" : '
                                             '23680,\n'
                                             '            "40.000000" : '
                                             '23936,\n'
                                             '            "50.000000" : '
                                             '24960,\n'
                                             '            "60.000000" : '
                                             '25984,\n'
                                             '            "70.000000" : '
                                             '26240,\n'
                                             '            "80.000000" : '
                                             '26496,\n'
                                             '            "90.000000" : '
                                             '26496,\n'
                                             '            "95.000000" : '
                                             '29824,\n'
                                             '            "99.000000" : '
                                             '38144,\n'
                                             '            "99.500000" : '
                                             '42752,\n'
                                             '            "99.900000" : '
                                             '52480,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 78336\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 5811,\n'
                                             '          "max" : 141034,\n'
                                             '          "mean" : '
                                             '27591.452594,\n'
                                             '          "stddev" : '
                                             '2976.337309,\n'
                                             '          "N" : 1975902\n'
                                             '        },\n'
                                             '        "bw_min" : 140763,\n'
                                             '        "bw_max" : 141655,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '141148.214286,\n'
                                             '        "bw_dev" : 223.325281,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 281526,\n'
                                             '        "iops_max" : 283310,\n'
                                             '        "iops_mean" : '
                                             '282296.428571,\n'
                                             '        "iops_stddev" : '
                                             '446.650563,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.814286,\n'
                                             '      "sys_cpu" : 59.185714,\n'
                                             '      "ctx" : 8781,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999975,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 1.898886,\n'
                                             '        "50" : 97.964619,\n'
                                             '        "100" : 0.134015,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=144420: Wed '
                                             'Oct 26 04:53:37 2022\n'
                                             '  read: IOPS=282k, BW=138MiB/s '
                                             '(145MB/s)(965MiB/7001msec)\n'
                                             '    slat (nsec): min=1482, '
                                             'max=25537, avg=2298.56, '
                                             'stdev=931.38\n'
                                             '    clat (usec): min=4, max=139, '
                                             'avg=25.29, stdev= 3.16\n'
                                             '     lat (usec): min=5, max=141, '
                                             'avg=27.59, stdev= 2.98\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[18304],  '
                                             '5.00th=[23168], 10.00th=[23424], '
                                             '20.00th=[23680],\n'
                                             '     | 30.00th=[23680], '
                                             '40.00th=[23936], '
                                             '50.00th=[24960], '
                                             '60.00th=[25984],\n'
                                             '     | 70.00th=[26240], '
                                             '80.00th=[26496], '
                                             '90.00th=[26496], '
                                             '95.00th=[29824],\n'
                                             '     | 99.00th=[38144], '
                                             '99.50th=[42752], '
                                             '99.90th=[52480], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[78336]\n'
                                             '   bw (  KiB/s): min=140763, '
                                             'max=141655, per=100.00%, '
                                             'avg=141148.21, stdev=223.33, '
                                             'samples=14\n'
                                             '   iops        : min=281526, '
                                             'max=283310, avg=282296.43, '
                                             'stdev=446.65, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=1.90%, 50=97.96%, 100=0.13%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=39.81%, '
                                             'sys=59.19%, ctx=8781, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1975895,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=138MiB/s (145MB/s), '
                                             '138MiB/s-138MiB/s '
                                             '(145MB/s-145MB/s), io=965MiB '
                                             '(1012MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.output'),
                                   'state': {'begin': 1666745728.1251886,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_512_8_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7885286808013916,
                                             'end': 1666745728.9137173,
                                             'err': 0,
                                             'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                             'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.output'},
                                   'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_396.state')},
                       'cmd_40': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.output'),
                                  'state': {'begin': 1666744023.1586106,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7830297946929932,
                                            'end': 1666744023.9416404,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_40.state')},
                       'cmd_41': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.output'),
                                  'state': {'begin': 1666744023.943622,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_uring" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.344534397125244,
                                            'end': 1666744035.2881565,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_41.state')},
                       'cmd_42': {'output': 'io_uring-bdev_uring: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751124,\n'
                                            '  "timestamp_ms" : '
                                            '1666751124416,\n'
                                            '  "time" : "Wed Oct 26 04:25:24 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_uring",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_uring",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 554997760,\n'
                                            '        "io_kbytes" : 541990,\n'
                                            '        "bw_bytes" : 79285394,\n'
                                            '        "bw" : 77427,\n'
                                            '        "iops" : 154854.285714,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 1083980,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 160,\n'
                                            '          "max" : 12163,\n'
                                            '          "mean" : 182.966398,\n'
                                            '          "stddev" : 38.361146,\n'
                                            '          "N" : 1083980\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 5690,\n'
                                            '          "max" : 79259,\n'
                                            '          "mean" : 6045.423559,\n'
                                            '          "stddev" : '
                                            '1095.010971,\n'
                                            '          "N" : 1083980,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 5792,\n'
                                            '            "5.000000" : 5792,\n'
                                            '            "10.000000" : 5792,\n'
                                            '            "20.000000" : 5856,\n'
                                            '            "30.000000" : 5856,\n'
                                            '            "40.000000" : 5856,\n'
                                            '            "50.000000" : 5920,\n'
                                            '            "60.000000" : 5920,\n'
                                            '            "70.000000" : 5920,\n'
                                            '            "80.000000" : 5984,\n'
                                            '            "90.000000" : 6048,\n'
                                            '            "95.000000" : 6176,\n'
                                            '            "99.000000" : 11456,\n'
                                            '            "99.500000" : 12096,\n'
                                            '            "99.900000" : 17024,\n'
                                            '            "99.950000" : 17024,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5871,\n'
                                            '          "max" : 79429,\n'
                                            '          "mean" : 6228.389957,\n'
                                            '          "stddev" : '
                                            '1095.955735,\n'
                                            '          "N" : 1083980\n'
                                            '        },\n'
                                            '        "bw_min" : 77263,\n'
                                            '        "bw_max" : 77571,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '77436.857143,\n'
                                            '        "bw_dev" : 79.987362,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 154526,\n'
                                            '        "iops_max" : 155142,\n'
                                            '        "iops_mean" : '
                                            '154873.785714,\n'
                                            '        "iops_stddev" : '
                                            '160.033736,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 48.900000,\n'
                                            '      "sys_cpu" : 51.071429,\n'
                                            '      "ctx" : 23,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.581247,\n'
                                            '        "20" : 1.403808,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013100,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_uring: (groupid=0, '
                                            'jobs=1): err= 0: pid=139233: Wed '
                                            'Oct 26 04:25:24 2022\n'
                                            '  read: IOPS=155k, BW=75.6MiB/s '
                                            '(79.3MB/s)(529MiB/7000msec)\n'
                                            '    slat (nsec): min=160, '
                                            'max=12163, avg=182.97, '
                                            'stdev=38.36\n'
                                            '    clat (nsec): min=5690, '
                                            'max=79259, avg=6045.42, '
                                            'stdev=1095.01\n'
                                            '     lat (nsec): min=5871, '
                                            'max=79429, avg=6228.39, '
                                            'stdev=1095.96\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 5792],  5.00th=[ '
                                            '5792], 10.00th=[ 5792], 20.00th=[ '
                                            '5856],\n'
                                            '     | 30.00th=[ 5856], 40.00th=[ '
                                            '5856], 50.00th=[ 5920], 60.00th=[ '
                                            '5920],\n'
                                            '     | 70.00th=[ 5920], 80.00th=[ '
                                            '5984], 90.00th=[ 6048], 95.00th=[ '
                                            '6176],\n'
                                            '     | 99.00th=[11456], '
                                            '99.50th=[12096], 99.90th=[17024], '
                                            '99.95th=[17024],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=77263, '
                                            'max=77571, per=100.00%, '
                                            'avg=77436.86, stdev=79.99, '
                                            'samples=14\n'
                                            '   iops        : min=154526, '
                                            'max=155142, avg=154873.79, '
                                            'stdev=160.03, samples=14\n'
                                            '  lat (usec)   : 10=98.58%, '
                                            '20=1.40%, 50=0.01%, 100=0.01%\n'
                                            '  cpu          : usr=48.90%, '
                                            'sys=51.07%, ctx=23, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1083980,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=75.6MiB/s (79.3MB/s), '
                                            '75.6MiB/s-75.6MiB/s '
                                            '(79.3MB/s-79.3MB/s), io=529MiB '
                                            '(555MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.output'),
                                  'state': {'begin': 1666744035.290713,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7952075004577637,
                                            'end': 1666744036.0859206,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_42.state')},
                       'cmd_43': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.output'),
                                  'state': {'begin': 1666744038.146734,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7901654243469238,
                                            'end': 1666744038.9368994,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_43.state')},
                       'cmd_44': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.output'),
                                  'state': {'begin': 1666744038.9389307,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_uring" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.368819952011108,
                                            'end': 1666744050.3077507,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_44.state')},
                       'cmd_45': {'output': 'io_uring-bdev_uring: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751139,\n'
                                            '  "timestamp_ms" : '
                                            '1666751139437,\n'
                                            '  "time" : "Wed Oct 26 04:25:39 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_uring",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_uring",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 556163072,\n'
                                            '        "io_kbytes" : 543128,\n'
                                            '        "bw_bytes" : 79440518,\n'
                                            '        "bw" : 77578,\n'
                                            '        "iops" : 155157.263248,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1086256,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 160,\n'
                                            '          "max" : 11702,\n'
                                            '          "mean" : 182.538771,\n'
                                            '          "stddev" : 44.114911,\n'
                                            '          "N" : 1086256\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 5681,\n'
                                            '          "max" : 114124,\n'
                                            '          "mean" : 6027.226244,\n'
                                            '          "stddev" : '
                                            '1126.443798,\n'
                                            '          "N" : 1086256,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 5728,\n'
                                            '            "5.000000" : 5792,\n'
                                            '            "10.000000" : 5792,\n'
                                            '            "20.000000" : 5792,\n'
                                            '            "30.000000" : 5792,\n'
                                            '            "40.000000" : 5856,\n'
                                            '            "50.000000" : 5856,\n'
                                            '            "60.000000" : 5920,\n'
                                            '            "70.000000" : 5920,\n'
                                            '            "80.000000" : 5920,\n'
                                            '            "90.000000" : 6048,\n'
                                            '            "95.000000" : 6112,\n'
                                            '            "99.000000" : 11456,\n'
                                            '            "99.500000" : 12736,\n'
                                            '            "99.900000" : 17024,\n'
                                            '            "99.950000" : 17024,\n'
                                            '            "99.990000" : 58624\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5861,\n'
                                            '          "max" : 114305,\n'
                                            '          "mean" : 6209.765015,\n'
                                            '          "stddev" : '
                                            '1127.684770,\n'
                                            '          "N" : 1086256\n'
                                            '        },\n'
                                            '        "bw_min" : 77442,\n'
                                            '        "bw_max" : 77945,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '77599.142857,\n'
                                            '        "bw_dev" : 130.531374,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 154884,\n'
                                            '        "iops_max" : 155890,\n'
                                            '        "iops_mean" : '
                                            '155198.357143,\n'
                                            '        "iops_stddev" : '
                                            '261.172388,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 47.071429,\n'
                                            '      "sys_cpu" : 52.900000,\n'
                                            '      "ctx" : 18,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 98.564611,\n'
                                            '        "20" : 1.419371,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013809,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_uring: (groupid=0, '
                                            'jobs=1): err= 0: pid=139283: Wed '
                                            'Oct 26 04:25:39 2022\n'
                                            '  read: IOPS=155k, BW=75.8MiB/s '
                                            '(79.4MB/s)(530MiB/7001msec)\n'
                                            '    slat (nsec): min=160, '
                                            'max=11702, avg=182.54, '
                                            'stdev=44.11\n'
                                            '    clat (usec): min=5, max=114, '
                                            'avg= 6.03, stdev= 1.13\n'
                                            '     lat (usec): min=5, max=114, '
                                            'avg= 6.21, stdev= 1.13\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 5728],  5.00th=[ '
                                            '5792], 10.00th=[ 5792], 20.00th=[ '
                                            '5792],\n'
                                            '     | 30.00th=[ 5792], 40.00th=[ '
                                            '5856], 50.00th=[ 5856], 60.00th=[ '
                                            '5920],\n'
                                            '     | 70.00th=[ 5920], 80.00th=[ '
                                            '5920], 90.00th=[ 6048], 95.00th=[ '
                                            '6112],\n'
                                            '     | 99.00th=[11456], '
                                            '99.50th=[12736], 99.90th=[17024], '
                                            '99.95th=[17024],\n'
                                            '     | 99.99th=[58624]\n'
                                            '   bw (  KiB/s): min=77442, '
                                            'max=77945, per=100.00%, '
                                            'avg=77599.14, stdev=130.53, '
                                            'samples=14\n'
                                            '   iops        : min=154884, '
                                            'max=155890, avg=155198.36, '
                                            'stdev=261.17, samples=14\n'
                                            '  lat (usec)   : 10=98.56%, '
                                            '20=1.42%, 50=0.01%, 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=47.07%, '
                                            'sys=52.90%, ctx=18, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1086256,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=75.8MiB/s (79.4MB/s), '
                                            '75.8MiB/s-75.8MiB/s '
                                            '(79.4MB/s-79.4MB/s), io=530MiB '
                                            '(556MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.output'),
                                  'state': {'begin': 1666744050.3104663,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7914083003997803,
                                            'end': 1666744051.1018746,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_45.state')},
                       'cmd_46': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.output'),
                                  'state': {'begin': 1666744053.1546078,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7848234176635742,
                                            'end': 1666744053.9394312,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_46.state')},
                       'cmd_47': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.output'),
                                  'state': {'begin': 1666744053.9416974,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.334166526794434,
                                            'end': 1666744065.275864,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_47.state')},
                       'cmd_48': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751154,\n'
                                            '  "timestamp_ms" : '
                                            '1666751154404,\n'
                                            '  "time" : "Wed Oct 26 04:25:54 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 579297280,\n'
                                            '        "io_kbytes" : 565720,\n'
                                            '        "bw_bytes" : 82744933,\n'
                                            '        "bw" : 80805,\n'
                                            '        "iops" : 161611.055563,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1131439,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1573,\n'
                                            '          "max" : 22613,\n'
                                            '          "mean" : 1629.937999,\n'
                                            '          "stddev" : 110.064556,\n'
                                            '          "N" : 1131440\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 231,\n'
                                            '          "max" : 100949,\n'
                                            '          "mean" : 4328.043267,\n'
                                            '          "stddev" : '
                                            '1152.033943,\n'
                                            '          "N" : 1131439,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4048,\n'
                                            '            "5.000000" : 4080,\n'
                                            '            "10.000000" : 4080,\n'
                                            '            "20.000000" : 4128,\n'
                                            '            "30.000000" : 4128,\n'
                                            '            "40.000000" : 4128,\n'
                                            '            "50.000000" : 4192,\n'
                                            '            "60.000000" : 4192,\n'
                                            '            "70.000000" : 4256,\n'
                                            '            "80.000000" : 4256,\n'
                                            '            "90.000000" : 4320,\n'
                                            '            "95.000000" : 4448,\n'
                                            '            "99.000000" : 10048,\n'
                                            '            "99.500000" : 10688,\n'
                                            '            "99.900000" : 15424,\n'
                                            '            "99.950000" : 15552,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5620,\n'
                                            '          "max" : 102592,\n'
                                            '          "mean" : 5957.981281,\n'
                                            '          "stddev" : '
                                            '1158.044008,\n'
                                            '          "N" : 1131439\n'
                                            '        },\n'
                                            '        "bw_min" : 80689,\n'
                                            '        "bw_max" : 80965,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '80824.285714,\n'
                                            '        "bw_dev" : 84.986360,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 161378,\n'
                                            '        "iops_max" : 161930,\n'
                                            '        "iops_mean" : '
                                            '161648.642857,\n'
                                            '        "iops_stddev" : '
                                            '169.934651,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 10.557143,\n'
                                            '      "sys_cpu" : 89.414286,\n'
                                            '      "ctx" : 24,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.957080,\n'
                                            '        "20" : 1.019763,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.014495,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=139333: Wed '
                                            'Oct 26 04:25:54 2022\n'
                                            '  read: IOPS=162k, BW=78.9MiB/s '
                                            '(82.7MB/s)(552MiB/7001msec)\n'
                                            '    slat (nsec): min=1573, '
                                            'max=22613, avg=1629.94, '
                                            'stdev=110.06\n'
                                            '    clat (nsec): min=231, '
                                            'max=100949, avg=4328.04, '
                                            'stdev=1152.03\n'
                                            '     lat (usec): min=5, max=102, '
                                            'avg= 5.96, stdev= 1.16\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4048],  5.00th=[ '
                                            '4080], 10.00th=[ 4080], 20.00th=[ '
                                            '4128],\n'
                                            '     | 30.00th=[ 4128], 40.00th=[ '
                                            '4128], 50.00th=[ 4192], 60.00th=[ '
                                            '4192],\n'
                                            '     | 70.00th=[ 4256], 80.00th=[ '
                                            '4256], 90.00th=[ 4320], 95.00th=[ '
                                            '4448],\n'
                                            '     | 99.00th=[10048], '
                                            '99.50th=[10688], 99.90th=[15424], '
                                            '99.95th=[15552],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=80689, '
                                            'max=80965, per=100.00%, '
                                            'avg=80824.29, stdev=84.99, '
                                            'samples=14\n'
                                            '   iops        : min=161378, '
                                            'max=161930, avg=161648.64, '
                                            'stdev=169.93, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%, 750=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=98.96%, 20=1.02%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=10.56%, '
                                            'sys=89.41%, ctx=24, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1131439,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=78.9MiB/s (82.7MB/s), '
                                            '78.9MiB/s-78.9MiB/s '
                                            '(82.7MB/s-82.7MB/s), io=552MiB '
                                            '(579MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.output'),
                                  'state': {'begin': 1666744065.2787638,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.797447681427002,
                                            'end': 1666744066.0762115,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_48.state')},
                       'cmd_49': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.output'),
                                  'state': {'begin': 1666744068.102742,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.789142370223999,
                                            'end': 1666744068.8918843,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_49.state')},
                       'cmd_50': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.output'),
                                  'state': {'begin': 1666744068.8940985,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.456218719482422,
                                            'end': 1666744080.3503172,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_50.state')},
                       'cmd_51': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751169,\n'
                                            '  "timestamp_ms" : '
                                            '1666751169479,\n'
                                            '  "time" : "Wed Oct 26 04:26:09 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 578703872,\n'
                                            '        "io_kbytes" : 565140,\n'
                                            '        "bw_bytes" : 82671981,\n'
                                            '        "bw" : 80734,\n'
                                            '        "iops" : 161468.714286,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 1130281,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1563,\n'
                                            '          "max" : 15979,\n'
                                            '          "mean" : 1627.234484,\n'
                                            '          "stddev" : 111.403997,\n'
                                            '          "N" : 1130281\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 230,\n'
                                            '          "max" : 75993,\n'
                                            '          "mean" : 4333.185862,\n'
                                            '          "stddev" : '
                                            '1142.444879,\n'
                                            '          "N" : 1130281,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4080,\n'
                                            '            "5.000000" : 4080,\n'
                                            '            "10.000000" : 4128,\n'
                                            '            "20.000000" : 4128,\n'
                                            '            "30.000000" : 4128,\n'
                                            '            "40.000000" : 4192,\n'
                                            '            "50.000000" : 4192,\n'
                                            '            "60.000000" : 4192,\n'
                                            '            "70.000000" : 4256,\n'
                                            '            "80.000000" : 4256,\n'
                                            '            "90.000000" : 4320,\n'
                                            '            "95.000000" : 4448,\n'
                                            '            "99.000000" : 10048,\n'
                                            '            "99.500000" : 11200,\n'
                                            '            "99.900000" : 15424,\n'
                                            '            "99.950000" : 15552,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5630,\n'
                                            '          "max" : 77626,\n'
                                            '          "mean" : 5960.420346,\n'
                                            '          "stddev" : '
                                            '1148.508228,\n'
                                            '          "N" : 1130281\n'
                                            '        },\n'
                                            '        "bw_min" : 80670,\n'
                                            '        "bw_max" : 80870,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '80741.785714,\n'
                                            '        "bw_dev" : 54.574125,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 161340,\n'
                                            '        "iops_max" : 161740,\n'
                                            '        "iops_mean" : '
                                            '161483.571429,\n'
                                            '        "iops_stddev" : '
                                            '109.148251,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 11.014286,\n'
                                            '      "sys_cpu" : 88.957143,\n'
                                            '      "ctx" : 20,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.965390,\n'
                                            '        "20" : 1.013199,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.012740,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=139382: Wed '
                                            'Oct 26 04:26:09 2022\n'
                                            '  read: IOPS=161k, BW=78.8MiB/s '
                                            '(82.7MB/s)(552MiB/7000msec)\n'
                                            '    slat (nsec): min=1563, '
                                            'max=15979, avg=1627.23, '
                                            'stdev=111.40\n'
                                            '    clat (nsec): min=230, '
                                            'max=75993, avg=4333.19, '
                                            'stdev=1142.44\n'
                                            '     lat (nsec): min=5630, '
                                            'max=77626, avg=5960.42, '
                                            'stdev=1148.51\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4080],  5.00th=[ '
                                            '4080], 10.00th=[ 4128], 20.00th=[ '
                                            '4128],\n'
                                            '     | 30.00th=[ 4128], 40.00th=[ '
                                            '4192], 50.00th=[ 4192], 60.00th=[ '
                                            '4192],\n'
                                            '     | 70.00th=[ 4256], 80.00th=[ '
                                            '4256], 90.00th=[ 4320], 95.00th=[ '
                                            '4448],\n'
                                            '     | 99.00th=[10048], '
                                            '99.50th=[11200], 99.90th=[15424], '
                                            '99.95th=[15552],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=80670, '
                                            'max=80870, per=100.00%, '
                                            'avg=80741.79, stdev=54.57, '
                                            'samples=14\n'
                                            '   iops        : min=161340, '
                                            'max=161740, avg=161483.57, '
                                            'stdev=109.15, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%, 750=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=98.97%, 20=1.01%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=11.01%, '
                                            'sys=88.96%, ctx=20, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1130281,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=78.8MiB/s (82.7MB/s), '
                                            '78.8MiB/s-78.8MiB/s '
                                            '(82.7MB/s-82.7MB/s), io=552MiB '
                                            '(579MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.output'),
                                  'state': {'begin': 1666744080.3534555,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7853691577911377,
                                            'end': 1666744081.1388247,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_51.state')},
                       'cmd_52': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.output'),
                                  'state': {'begin': 1666744083.15069,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7832794189453125,
                                            'end': 1666744083.9339695,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_52.state')},
                       'cmd_53': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.output'),
                                  'state': {'begin': 1666744083.9360201,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.43041706085205,
                                            'end': 1666744095.3664372,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_53.state')},
                       'cmd_54': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751184,\n'
                                            '  "timestamp_ms" : '
                                            '1666751184500,\n'
                                            '  "time" : "Wed Oct 26 04:26:24 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 576747520,\n'
                                            '        "io_kbytes" : 563230,\n'
                                            '        "bw_bytes" : 82380734,\n'
                                            '        "bw" : 80449,\n'
                                            '        "iops" : 160899.871447,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1126460,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1563,\n'
                                            '          "max" : 15619,\n'
                                            '          "mean" : 1623.509082,\n'
                                            '          "stddev" : 104.723698,\n'
                                            '          "N" : 1126460\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 241,\n'
                                            '          "max" : 77606,\n'
                                            '          "mean" : 4359.351258,\n'
                                            '          "stddev" : '
                                            '1122.867630,\n'
                                            '          "N" : 1126460,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4128,\n'
                                            '            "5.000000" : 4128,\n'
                                            '            "10.000000" : 4128,\n'
                                            '            "20.000000" : 4128,\n'
                                            '            "30.000000" : 4128,\n'
                                            '            "40.000000" : 4192,\n'
                                            '            "50.000000" : 4192,\n'
                                            '            "60.000000" : 4256,\n'
                                            '            "70.000000" : 4256,\n'
                                            '            "80.000000" : 4320,\n'
                                            '            "90.000000" : 4384,\n'
                                            '            "95.000000" : 4448,\n'
                                            '            "99.000000" : 10048,\n'
                                            '            "99.500000" : 10688,\n'
                                            '            "99.900000" : 15424,\n'
                                            '            "99.950000" : 15552,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5650,\n'
                                            '          "max" : 79209,\n'
                                            '          "mean" : 5982.860339,\n'
                                            '          "stddev" : '
                                            '1128.826196,\n'
                                            '          "N" : 1126460\n'
                                            '        },\n'
                                            '        "bw_min" : 80316,\n'
                                            '        "bw_max" : 80630,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '80486.615385,\n'
                                            '        "bw_dev" : 102.524419,\n'
                                            '        "bw_samples" : 13,\n'
                                            '        "iops_min" : 160632,\n'
                                            '        "iops_max" : 161260,\n'
                                            '        "iops_mean" : '
                                            '160973.230769,\n'
                                            '        "iops_stddev" : '
                                            '205.048837,\n'
                                            '        "iops_samples" : 13\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 10.328571,\n'
                                            '      "sys_cpu" : 89.642857,\n'
                                            '      "ctx" : 21,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.971468,\n'
                                            '        "20" : 1.008291,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013050,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=139431: Wed '
                                            'Oct 26 04:26:24 2022\n'
                                            '  read: IOPS=161k, BW=78.6MiB/s '
                                            '(82.4MB/s)(550MiB/7001msec)\n'
                                            '    slat (nsec): min=1563, '
                                            'max=15619, avg=1623.51, '
                                            'stdev=104.72\n'
                                            '    clat (nsec): min=241, '
                                            'max=77606, avg=4359.35, '
                                            'stdev=1122.87\n'
                                            '     lat (nsec): min=5650, '
                                            'max=79209, avg=5982.86, '
                                            'stdev=1128.83\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4128],  5.00th=[ '
                                            '4128], 10.00th=[ 4128], 20.00th=[ '
                                            '4128],\n'
                                            '     | 30.00th=[ 4128], 40.00th=[ '
                                            '4192], 50.00th=[ 4192], 60.00th=[ '
                                            '4256],\n'
                                            '     | 70.00th=[ 4256], 80.00th=[ '
                                            '4320], 90.00th=[ 4384], 95.00th=[ '
                                            '4448],\n'
                                            '     | 99.00th=[10048], '
                                            '99.50th=[10688], 99.90th=[15424], '
                                            '99.95th=[15552],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=80316, '
                                            'max=80630, per=100.00%, '
                                            'avg=80486.62, stdev=102.52, '
                                            'samples=13\n'
                                            '   iops        : min=160632, '
                                            'max=161260, avg=160973.23, '
                                            'stdev=205.05, samples=13\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%, 750=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=98.97%, 20=1.01%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=10.33%, '
                                            'sys=89.64%, ctx=21, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1126460,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=78.6MiB/s (82.4MB/s), '
                                            '78.6MiB/s-78.6MiB/s '
                                            '(82.4MB/s-82.4MB/s), io=550MiB '
                                            '(577MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.output'),
                                  'state': {'begin': 1666744095.3690445,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7837162017822266,
                                            'end': 1666744096.1527607,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_54.state')},
                       'cmd_55': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.output'),
                                  'state': {'begin': 1666744097.1452632,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7812321186065674,
                                            'end': 1666744097.9264953,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_55.state')},
                       'cmd_56': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.output'),
                                  'state': {'begin': 1666744097.9284656,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.052183866500854,
                                            'end': 1666744108.9806495,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_56.state')},
                       'cmd_57': {'output': 'io_uring-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751198,\n'
                                            '  "timestamp_ms" : '
                                            '1666751198179,\n'
                                            '  "time" : "Wed Oct 26 04:26:38 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 278675456,\n'
                                            '        "io_kbytes" : 272144,\n'
                                            '        "bw_bytes" : 39805092,\n'
                                            '        "bw" : 38872,\n'
                                            '        "iops" : 77744.179403,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 544287,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2585,\n'
                                            '          "max" : 41207,\n'
                                            '          "mean" : 2875.072484,\n'
                                            '          "stddev" : 249.042554,\n'
                                            '          "N" : 544288\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 301,\n'
                                            '          "max" : 155822,\n'
                                            '          "mean" : 9423.841889,\n'
                                            '          "stddev" : '
                                            '1236.552305,\n'
                                            '          "N" : 544287,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 9024,\n'
                                            '            "5.000000" : 9024,\n'
                                            '            "10.000000" : 9152,\n'
                                            '            "20.000000" : 9152,\n'
                                            '            "30.000000" : 9280,\n'
                                            '            "40.000000" : 9280,\n'
                                            '            "50.000000" : 9280,\n'
                                            '            "60.000000" : 9280,\n'
                                            '            "70.000000" : 9280,\n'
                                            '            "80.000000" : 9408,\n'
                                            '            "90.000000" : 9408,\n'
                                            '            "95.000000" : 9664,\n'
                                            '            "99.000000" : 15168,\n'
                                            '            "99.500000" : 15808,\n'
                                            '            "99.900000" : 19840,\n'
                                            '            "99.950000" : 26752,\n'
                                            '            "99.990000" : 55552\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9348,\n'
                                            '          "max" : 158678,\n'
                                            '          "mean" : 12298.914446,\n'
                                            '          "stddev" : '
                                            '1258.957845,\n'
                                            '          "N" : 544287\n'
                                            '        },\n'
                                            '        "bw_min" : 38691,\n'
                                            '        "bw_max" : 39063,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '38882.357143,\n'
                                            '        "bw_dev" : 110.504860,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 77382,\n'
                                            '        "iops_max" : 78126,\n'
                                            '        "iops_mean" : '
                                            '77764.714286,\n'
                                            '        "iops_stddev" : '
                                            '221.009721,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 13.557143,\n'
                                            '      "sys_cpu" : 47.928571,\n'
                                            '      "ctx" : 544144,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.011942,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 96.766412,\n'
                                            '        "20" : 3.138785,\n'
                                            '        "50" : 0.065958,\n'
                                            '        "100" : 0.016535,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=139470: Wed '
                                            'Oct 26 04:26:38 2022\n'
                                            '  read: IOPS=77.7k, BW=38.0MiB/s '
                                            '(39.8MB/s)(266MiB/7001msec)\n'
                                            '    slat (nsec): min=2585, '
                                            'max=41207, avg=2875.07, '
                                            'stdev=249.04\n'
                                            '    clat (nsec): min=301, '
                                            'max=155822, avg=9423.84, '
                                            'stdev=1236.55\n'
                                            '     lat (usec): min=9, max=158, '
                                            'avg=12.30, stdev= 1.26\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 9024],  5.00th=[ '
                                            '9024], 10.00th=[ 9152], 20.00th=[ '
                                            '9152],\n'
                                            '     | 30.00th=[ 9280], 40.00th=[ '
                                            '9280], 50.00th=[ 9280], 60.00th=[ '
                                            '9280],\n'
                                            '     | 70.00th=[ 9280], 80.00th=[ '
                                            '9408], 90.00th=[ 9408], 95.00th=[ '
                                            '9664],\n'
                                            '     | 99.00th=[15168], '
                                            '99.50th=[15808], 99.90th=[19840], '
                                            '99.95th=[26752],\n'
                                            '     | 99.99th=[55552]\n'
                                            '   bw (  KiB/s): min=38691, '
                                            'max=39063, per=100.00%, '
                                            'avg=38882.36, stdev=110.50, '
                                            'samples=14\n'
                                            '   iops        : min=77382, '
                                            'max=78126, avg=77764.71, '
                                            'stdev=221.01, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%, '
                                            '750=0.01%\n'
                                            '  lat (usec)   : 10=96.77%, '
                                            '20=3.14%, 50=0.07%, 100=0.02%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=13.56%, '
                                            'sys=47.93%, ctx=544144, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=544287,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=38.0MiB/s (39.8MB/s), '
                                            '38.0MiB/s-38.0MiB/s '
                                            '(39.8MB/s-39.8MB/s), io=266MiB '
                                            '(279MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.output'),
                                  'state': {'begin': 1666744108.9835074,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7877578735351562,
                                            'end': 1666744109.7712653,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_57.state')},
                       'cmd_58': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.output'),
                                  'state': {'begin': 1666744110.759508,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7883443832397461,
                                            'end': 1666744111.5478523,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_58.state')},
                       'cmd_59': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.output'),
                                  'state': {'begin': 1666744111.5498905,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.059826374053955,
                                            'end': 1666744122.609717,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_59.state')},
                       'cmd_60': {'output': 'io_uring-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751211,\n'
                                            '  "timestamp_ms" : '
                                            '1666751211808,\n'
                                            '  "time" : "Wed Oct 26 04:26:51 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 277933056,\n'
                                            '        "io_kbytes" : 271419,\n'
                                            '        "bw_bytes" : 39699050,\n'
                                            '        "bw" : 38768,\n'
                                            '        "iops" : 77537.208970,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 542838,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2625,\n'
                                            '          "max" : 47760,\n'
                                            '          "mean" : 2836.992208,\n'
                                            '          "stddev" : 284.296938,\n'
                                            '          "N" : 542838\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 311,\n'
                                            '          "max" : 110006,\n'
                                            '          "mean" : 9445.447319,\n'
                                            '          "stddev" : '
                                            '1220.518444,\n'
                                            '          "N" : 542838,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 9024,\n'
                                            '            "5.000000" : 9152,\n'
                                            '            "10.000000" : 9152,\n'
                                            '            "20.000000" : 9152,\n'
                                            '            "30.000000" : 9280,\n'
                                            '            "40.000000" : 9280,\n'
                                            '            "50.000000" : 9280,\n'
                                            '            "60.000000" : 9280,\n'
                                            '            "70.000000" : 9408,\n'
                                            '            "80.000000" : 9408,\n'
                                            '            "90.000000" : 9408,\n'
                                            '            "95.000000" : 9664,\n'
                                            '            "99.000000" : 15168,\n'
                                            '            "99.500000" : 15808,\n'
                                            '            "99.900000" : 20096,\n'
                                            '            "99.950000" : 27264,\n'
                                            '            "99.990000" : 56064\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9437,\n'
                                            '          "max" : 112812,\n'
                                            '          "mean" : 12282.439527,\n'
                                            '          "stddev" : '
                                            '1251.244279,\n'
                                            '          "N" : 542838\n'
                                            '        },\n'
                                            '        "bw_min" : 38539,\n'
                                            '        "bw_max" : 38959,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '38780.285714,\n'
                                            '        "bw_dev" : 105.847807,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 77078,\n'
                                            '        "iops_max" : 77918,\n'
                                            '        "iops_mean" : '
                                            '77560.642857,\n'
                                            '        "iops_stddev" : '
                                            '211.751526,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 13.314286,\n'
                                            '      "sys_cpu" : 48.014286,\n'
                                            '      "ctx" : 542695,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010685,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 96.858547,\n'
                                            '        "20" : 3.030923,\n'
                                            '        "50" : 0.083819,\n'
                                            '        "100" : 0.015658,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=139510: Wed '
                                            'Oct 26 04:26:51 2022\n'
                                            '  read: IOPS=77.5k, BW=37.9MiB/s '
                                            '(39.7MB/s)(265MiB/7001msec)\n'
                                            '    slat (nsec): min=2625, '
                                            'max=47760, avg=2836.99, '
                                            'stdev=284.30\n'
                                            '    clat (nsec): min=311, '
                                            'max=110006, avg=9445.45, '
                                            'stdev=1220.52\n'
                                            '     lat (usec): min=9, max=112, '
                                            'avg=12.28, stdev= 1.25\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 9024],  5.00th=[ '
                                            '9152], 10.00th=[ 9152], 20.00th=[ '
                                            '9152],\n'
                                            '     | 30.00th=[ 9280], 40.00th=[ '
                                            '9280], 50.00th=[ 9280], 60.00th=[ '
                                            '9280],\n'
                                            '     | 70.00th=[ 9408], 80.00th=[ '
                                            '9408], 90.00th=[ 9408], 95.00th=[ '
                                            '9664],\n'
                                            '     | 99.00th=[15168], '
                                            '99.50th=[15808], 99.90th=[20096], '
                                            '99.95th=[27264],\n'
                                            '     | 99.99th=[56064]\n'
                                            '   bw (  KiB/s): min=38539, '
                                            'max=38959, per=100.00%, '
                                            'avg=38780.29, stdev=105.85, '
                                            'samples=14\n'
                                            '   iops        : min=77078, '
                                            'max=77918, avg=77560.64, '
                                            'stdev=211.75, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%, '
                                            '750=0.01%\n'
                                            '  lat (usec)   : 10=96.86%, '
                                            '20=3.03%, 50=0.08%, 100=0.02%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=13.31%, '
                                            'sys=48.01%, ctx=542695, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=542838,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=37.9MiB/s (39.7MB/s), '
                                            '37.9MiB/s-37.9MiB/s '
                                            '(39.7MB/s-39.7MB/s), io=265MiB '
                                            '(278MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.output'),
                                  'state': {'begin': 1666744122.6130521,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7879660129547119,
                                            'end': 1666744123.4010181,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_60.state')},
                       'cmd_61': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.output'),
                                  'state': {'begin': 1666744124.3926356,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7812538146972656,
                                            'end': 1666744125.1738894,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_61.state')},
                       'cmd_62': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.output'),
                                  'state': {'begin': 1666744125.1761858,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_io_uring-xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.060308694839478,
                                            'end': 1666744136.2364945,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_62.state')},
                       'cmd_63': {'output': 'io_uring-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751225,\n'
                                            '  "timestamp_ms" : '
                                            '1666751225436,\n'
                                            '  "time" : "Wed Oct 26 04:27:05 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 275073536,\n'
                                            '        "io_kbytes" : 268626,\n'
                                            '        "bw_bytes" : 39290606,\n'
                                            '        "bw" : 38369,\n'
                                            '        "iops" : 76739.465791,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 537253,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2655,\n'
                                            '          "max" : 33883,\n'
                                            '          "mean" : 2914.394468,\n'
                                            '          "stddev" : 191.365400,\n'
                                            '          "N" : 537253\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 301,\n'
                                            '          "max" : 98825,\n'
                                            '          "mean" : 9502.469501,\n'
                                            '          "stddev" : '
                                            '1206.508405,\n'
                                            '          "N" : 537253,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 9024,\n'
                                            '            "5.000000" : 9152,\n'
                                            '            "10.000000" : 9152,\n'
                                            '            "20.000000" : 9280,\n'
                                            '            "30.000000" : 9280,\n'
                                            '            "40.000000" : 9280,\n'
                                            '            "50.000000" : 9408,\n'
                                            '            "60.000000" : 9408,\n'
                                            '            "70.000000" : 9408,\n'
                                            '            "80.000000" : 9408,\n'
                                            '            "90.000000" : 9536,\n'
                                            '            "95.000000" : 9664,\n'
                                            '            "99.000000" : 15168,\n'
                                            '            "99.500000" : 15680,\n'
                                            '            "99.900000" : 19840,\n'
                                            '            "99.950000" : 25472,\n'
                                            '            "99.990000" : 57600\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9358,\n'
                                            '          "max" : 101720,\n'
                                            '          "mean" : 12416.863969,\n'
                                            '          "stddev" : '
                                            '1220.253756,\n'
                                            '          "N" : 537253\n'
                                            '        },\n'
                                            '        "bw_min" : 38246,\n'
                                            '        "bw_max" : 38615,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '38379.428571,\n'
                                            '        "bw_dev" : 100.427329,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 76492,\n'
                                            '        "iops_max" : 77230,\n'
                                            '        "iops_mean" : '
                                            '76758.857143,\n'
                                            '        "iops_stddev" : '
                                            '200.854657,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 15.200000,\n'
                                            '      "sys_cpu" : 46.171429,\n'
                                            '      "ctx" : 537127,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.011726,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 96.830916,\n'
                                            '        "20" : 3.088675,\n'
                                            '        "50" : 0.051372,\n'
                                            '        "100" : 0.017310,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=139549: Wed '
                                            'Oct 26 04:27:05 2022\n'
                                            '  read: IOPS=76.7k, BW=37.5MiB/s '
                                            '(39.3MB/s)(262MiB/7001msec)\n'
                                            '    slat (nsec): min=2655, '
                                            'max=33883, avg=2914.39, '
                                            'stdev=191.37\n'
                                            '    clat (nsec): min=301, '
                                            'max=98825, avg=9502.47, '
                                            'stdev=1206.51\n'
                                            '     lat (usec): min=9, max=101, '
                                            'avg=12.42, stdev= 1.22\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 9024],  5.00th=[ '
                                            '9152], 10.00th=[ 9152], 20.00th=[ '
                                            '9280],\n'
                                            '     | 30.00th=[ 9280], 40.00th=[ '
                                            '9280], 50.00th=[ 9408], 60.00th=[ '
                                            '9408],\n'
                                            '     | 70.00th=[ 9408], 80.00th=[ '
                                            '9408], 90.00th=[ 9536], 95.00th=[ '
                                            '9664],\n'
                                            '     | 99.00th=[15168], '
                                            '99.50th=[15680], 99.90th=[19840], '
                                            '99.95th=[25472],\n'
                                            '     | 99.99th=[57600]\n'
                                            '   bw (  KiB/s): min=38246, '
                                            'max=38615, per=100.00%, '
                                            'avg=38379.43, stdev=100.43, '
                                            'samples=14\n'
                                            '   iops        : min=76492, '
                                            'max=77230, avg=76758.86, '
                                            'stdev=200.85, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 10=96.83%, '
                                            '20=3.09%, 50=0.05%, 100=0.02%\n'
                                            '  cpu          : usr=15.20%, '
                                            'sys=46.17%, ctx=537127, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=537253,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=37.5MiB/s (39.3MB/s), '
                                            '37.5MiB/s-37.5MiB/s '
                                            '(39.3MB/s-39.3MB/s), io=262MiB '
                                            '(275MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.output'),
                                  'state': {'begin': 1666744136.2393565,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_io_uring-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7884199619293213,
                                            'end': 1666744137.0277765,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_63.state')},
                       'cmd_64': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.output'),
                                  'state': {'begin': 1666744138.0143127,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7859430313110352,
                                            'end': 1666744138.8002558,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_64.state')},
                       'cmd_65': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.output'),
                                  'state': {'begin': 1666744138.8029006,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="libaio" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-reference_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.079595804214478,
                                            'end': 1666744149.8824964,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_65.state')},
                       'cmd_66': {'output': 'libaio-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=libaio, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751239,\n'
                                            '  "timestamp_ms" : '
                                            '1666751239082,\n'
                                            '  "time" : "Wed Oct 26 04:27:19 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "libaio",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 285667328,\n'
                                            '        "io_kbytes" : 278972,\n'
                                            '        "bw_bytes" : 40803789,\n'
                                            '        "bw" : 39847,\n'
                                            '        "iops" : 79694.900728,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 557944,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2726,\n'
                                            '          "max" : 35376,\n'
                                            '          "mean" : 2949.894054,\n'
                                            '          "stddev" : 248.175186,\n'
                                            '          "N" : 557944\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 801,\n'
                                            '          "max" : 72345,\n'
                                            '          "mean" : 8997.065191,\n'
                                            '          "stddev" : '
                                            '1285.566522,\n'
                                            '          "N" : 557944,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8640,\n'
                                            '            "5.000000" : 8640,\n'
                                            '            "10.000000" : 8640,\n'
                                            '            "20.000000" : 8768,\n'
                                            '            "30.000000" : 8768,\n'
                                            '            "40.000000" : 8768,\n'
                                            '            "50.000000" : 8768,\n'
                                            '            "60.000000" : 8896,\n'
                                            '            "70.000000" : 8896,\n'
                                            '            "80.000000" : 8896,\n'
                                            '            "90.000000" : 9024,\n'
                                            '            "95.000000" : 9152,\n'
                                            '            "99.000000" : 15168,\n'
                                            '            "99.500000" : 15808,\n'
                                            '            "99.900000" : 19328,\n'
                                            '            "99.950000" : 26240,\n'
                                            '            "99.990000" : 55552\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9508,\n'
                                            '          "max" : 75291,\n'
                                            '          "mean" : 11946.959245,\n'
                                            '          "stddev" : '
                                            '1309.323831,\n'
                                            '          "N" : 557944\n'
                                            '        },\n'
                                            '        "bw_min" : 39689,\n'
                                            '        "bw_max" : 39936,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '39863.642857,\n'
                                            '        "bw_dev" : 67.919191,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 79378,\n'
                                            '        "iops_max" : 79872,\n'
                                            '        "iops_mean" : '
                                            '79727.357143,\n'
                                            '        "iops_stddev" : '
                                            '135.822060,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 14.842857,\n'
                                            '      "sys_cpu" : 45.585714,\n'
                                            '      "ctx" : 557800,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 97.375543,\n'
                                            '        "20" : 2.532512,\n'
                                            '        "50" : 0.062013,\n'
                                            '        "100" : 0.019536,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 799685,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5026,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5026,\n'
                                            '      "util" : 99.088748\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=139589: Wed '
                                            'Oct 26 04:27:19 2022\n'
                                            '  read: IOPS=79.7k, BW=38.9MiB/s '
                                            '(40.8MB/s)(272MiB/7001msec)\n'
                                            '    slat (nsec): min=2726, '
                                            'max=35376, avg=2949.89, '
                                            'stdev=248.18\n'
                                            '    clat (nsec): min=801, '
                                            'max=72345, avg=8997.07, '
                                            'stdev=1285.57\n'
                                            '     lat (nsec): min=9508, '
                                            'max=75291, avg=11946.96, '
                                            'stdev=1309.32\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8640],  5.00th=[ '
                                            '8640], 10.00th=[ 8640], 20.00th=[ '
                                            '8768],\n'
                                            '     | 30.00th=[ 8768], 40.00th=[ '
                                            '8768], 50.00th=[ 8768], 60.00th=[ '
                                            '8896],\n'
                                            '     | 70.00th=[ 8896], 80.00th=[ '
                                            '8896], 90.00th=[ 9024], 95.00th=[ '
                                            '9152],\n'
                                            '     | 99.00th=[15168], '
                                            '99.50th=[15808], 99.90th=[19328], '
                                            '99.95th=[26240],\n'
                                            '     | 99.99th=[55552]\n'
                                            '   bw (  KiB/s): min=39689, '
                                            'max=39936, per=100.00%, '
                                            'avg=39863.64, stdev=67.92, '
                                            'samples=14\n'
                                            '   iops        : min=79378, '
                                            'max=79872, avg=79727.36, '
                                            'stdev=135.82, samples=14\n'
                                            '  lat (nsec)   : 1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '10=97.38%, 20=2.53%, 50=0.06%, '
                                            '100=0.02%\n'
                                            '  cpu          : usr=14.84%, '
                                            'sys=45.59%, ctx=557800, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=557944,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=38.9MiB/s (40.8MB/s), '
                                            '38.9MiB/s-38.9MiB/s '
                                            '(40.8MB/s-40.8MB/s), io=272MiB '
                                            '(286MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=799685/0, '
                                            'merge=0/0, ticks=5026/0, '
                                            'in_queue=5026, util=99.09%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.output'),
                                  'state': {'begin': 1666744149.8855157,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7835009098052979,
                                            'end': 1666744150.6690166,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_66.state')},
                       'cmd_67': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.output'),
                                  'state': {'begin': 1666744151.6574724,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7808341979980469,
                                            'end': 1666744152.4383066,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_67.state')},
                       'cmd_68': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.output'),
                                  'state': {'begin': 1666744152.440717,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="libaio" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-reference_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.063346147537231,
                                            'end': 1666744163.5040631,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_68.state')},
                       'cmd_69': {'output': 'libaio-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=libaio, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751252,\n'
                                            '  "timestamp_ms" : '
                                            '1666751252703,\n'
                                            '  "time" : "Wed Oct 26 04:27:32 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "libaio",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 435968000,\n'
                                            '        "io_kbytes" : 425750,\n'
                                            '        "bw_bytes" : 62272246,\n'
                                            '        "bw" : 60812,\n'
                                            '        "iops" : 121625.482074,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 851500,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1543,\n'
                                            '          "max" : 14407,\n'
                                            '          "mean" : 1658.127843,\n'
                                            '          "stddev" : 52.610402,\n'
                                            '          "N" : 851500\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 410,\n'
                                            '          "max" : 96491,\n'
                                            '          "mean" : 6259.916447,\n'
                                            '          "stddev" : '
                                            '1052.788853,\n'
                                            '          "N" : 851500,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 5920,\n'
                                            '            "5.000000" : 5984,\n'
                                            '            "10.000000" : 5984,\n'
                                            '            "20.000000" : 6048,\n'
                                            '            "30.000000" : 6048,\n'
                                            '            "40.000000" : 6112,\n'
                                            '            "50.000000" : 6112,\n'
                                            '            "60.000000" : 6176,\n'
                                            '            "70.000000" : 6176,\n'
                                            '            "80.000000" : 6240,\n'
                                            '            "90.000000" : 6368,\n'
                                            '            "95.000000" : 6432,\n'
                                            '            "99.000000" : 11968,\n'
                                            '            "99.500000" : 12480,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15680,\n'
                                            '            "99.990000" : 56064\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 6091,\n'
                                            '          "max" : 98144,\n'
                                            '          "mean" : 7918.044290,\n'
                                            '          "stddev" : '
                                            '1054.920677,\n'
                                            '          "N" : 851500\n'
                                            '        },\n'
                                            '        "bw_min" : 60701,\n'
                                            '        "bw_max" : 61033,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '60838.642857,\n'
                                            '        "bw_dev" : 110.521566,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 121402,\n'
                                            '        "iops_max" : 122066,\n'
                                            '        "iops_mean" : '
                                            '121677.428571,\n'
                                            '        "iops_stddev" : '
                                            '221.098208,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 13.700000,\n'
                                            '      "sys_cpu" : 36.571429,\n'
                                            '      "ctx" : 851445,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.463065,\n'
                                            '        "20" : 1.515326,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.011861,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 1221975,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5983,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5983,\n'
                                            '      "util" : 99.088748\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=139627: Wed '
                                            'Oct 26 04:27:32 2022\n'
                                            '  read: IOPS=122k, BW=59.4MiB/s '
                                            '(62.3MB/s)(416MiB/7001msec)\n'
                                            '    slat (nsec): min=1543, '
                                            'max=14407, avg=1658.13, '
                                            'stdev=52.61\n'
                                            '    clat (nsec): min=410, '
                                            'max=96491, avg=6259.92, '
                                            'stdev=1052.79\n'
                                            '     lat (nsec): min=6091, '
                                            'max=98144, avg=7918.04, '
                                            'stdev=1054.92\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 5920],  5.00th=[ '
                                            '5984], 10.00th=[ 5984], 20.00th=[ '
                                            '6048],\n'
                                            '     | 30.00th=[ 6048], 40.00th=[ '
                                            '6112], 50.00th=[ 6112], 60.00th=[ '
                                            '6176],\n'
                                            '     | 70.00th=[ 6176], 80.00th=[ '
                                            '6240], 90.00th=[ 6368], 95.00th=[ '
                                            '6432],\n'
                                            '     | 99.00th=[11968], '
                                            '99.50th=[12480], 99.90th=[15552], '
                                            '99.95th=[15680],\n'
                                            '     | 99.99th=[56064]\n'
                                            '   bw (  KiB/s): min=60701, '
                                            'max=61033, per=100.00%, '
                                            'avg=60838.64, stdev=110.52, '
                                            'samples=14\n'
                                            '   iops        : min=121402, '
                                            'max=122066, avg=121677.43, '
                                            'stdev=221.10, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%, '
                                            '750=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=98.46%, 20=1.52%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=13.70%, '
                                            'sys=36.57%, ctx=851445, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=851500,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=59.4MiB/s (62.3MB/s), '
                                            '59.4MiB/s-59.4MiB/s '
                                            '(62.3MB/s-62.3MB/s), io=416MiB '
                                            '(436MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=1221975/0, '
                                            'merge=0/0, ticks=5983/0, '
                                            'in_queue=5983, util=99.09%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.output'),
                                  'state': {'begin': 1666744163.5061615,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7857673168182373,
                                            'end': 1666744164.2919288,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_69.state')},
                       'cmd_70': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.output'),
                                  'state': {'begin': 1666744165.280862,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7872707843780518,
                                            'end': 1666744166.0681329,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_70.state')},
                       'cmd_71': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.output'),
                                  'state': {'begin': 1666744166.0701516,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="libaio" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-reference_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.097615957260132,
                                            'end': 1666744177.1677675,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_71.state')},
                       'cmd_72': {'output': 'libaio-reference: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=libaio, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751266,\n'
                                            '  "timestamp_ms" : '
                                            '1666751266367,\n'
                                            '  "time" : "Wed Oct 26 04:27:46 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "libaio",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 281868800,\n'
                                            '        "io_kbytes" : 275262,\n'
                                            '        "bw_bytes" : 40261219,\n'
                                            '        "bw" : 39317,\n'
                                            '        "iops" : 78635.194972,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 550525,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2765,\n'
                                            '          "max" : 34235,\n'
                                            '          "mean" : 2952.005994,\n'
                                            '          "stddev" : 196.975536,\n'
                                            '          "N" : 550525\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 791,\n'
                                            '          "max" : 76494,\n'
                                            '          "mean" : 9169.578695,\n'
                                            '          "stddev" : '
                                            '1235.513611,\n'
                                            '          "N" : 550525,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8768,\n'
                                            '            "5.000000" : 8768,\n'
                                            '            "10.000000" : 8896,\n'
                                            '            "20.000000" : 8896,\n'
                                            '            "30.000000" : 8896,\n'
                                            '            "40.000000" : 9024,\n'
                                            '            "50.000000" : 9024,\n'
                                            '            "60.000000" : 9024,\n'
                                            '            "70.000000" : 9024,\n'
                                            '            "80.000000" : 9152,\n'
                                            '            "90.000000" : 9152,\n'
                                            '            "95.000000" : 9280,\n'
                                            '            "99.000000" : 15040,\n'
                                            '            "99.500000" : 15680,\n'
                                            '            "99.900000" : 19328,\n'
                                            '            "99.950000" : 23936,\n'
                                            '            "99.990000" : 57600\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9348,\n'
                                            '          "max" : 79429,\n'
                                            '          "mean" : 12121.584689,\n'
                                            '          "stddev" : '
                                            '1250.717136,\n'
                                            '          "N" : 550525\n'
                                            '        },\n'
                                            '        "bw_min" : 39172,\n'
                                            '        "bw_max" : 39524,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '39335.142857,\n'
                                            '        "bw_dev" : 102.275538,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 78344,\n'
                                            '        "iops_max" : 79048,\n'
                                            '        "iops_mean" : '
                                            '78670.428571,\n'
                                            '        "iops_stddev" : '
                                            '204.625630,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 16.157143,\n'
                                            '      "sys_cpu" : 44.100000,\n'
                                            '      "ctx" : 550395,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 97.186867,\n'
                                            '        "20" : 2.744653,\n'
                                            '        "50" : 0.040507,\n'
                                            '        "100" : 0.018709,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 790291,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5056,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5056,\n'
                                            '      "util" : 99.058940\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=139665: Wed '
                                            'Oct 26 04:27:46 2022\n'
                                            '  read: IOPS=78.6k, BW=38.4MiB/s '
                                            '(40.3MB/s)(269MiB/7001msec)\n'
                                            '    slat (nsec): min=2765, '
                                            'max=34235, avg=2952.01, '
                                            'stdev=196.98\n'
                                            '    clat (nsec): min=791, '
                                            'max=76494, avg=9169.58, '
                                            'stdev=1235.51\n'
                                            '     lat (nsec): min=9348, '
                                            'max=79429, avg=12121.58, '
                                            'stdev=1250.72\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8768],  5.00th=[ '
                                            '8768], 10.00th=[ 8896], 20.00th=[ '
                                            '8896],\n'
                                            '     | 30.00th=[ 8896], 40.00th=[ '
                                            '9024], 50.00th=[ 9024], 60.00th=[ '
                                            '9024],\n'
                                            '     | 70.00th=[ 9024], 80.00th=[ '
                                            '9152], 90.00th=[ 9152], 95.00th=[ '
                                            '9280],\n'
                                            '     | 99.00th=[15040], '
                                            '99.50th=[15680], 99.90th=[19328], '
                                            '99.95th=[23936],\n'
                                            '     | 99.99th=[57600]\n'
                                            '   bw (  KiB/s): min=39172, '
                                            'max=39524, per=100.00%, '
                                            'avg=39335.14, stdev=102.28, '
                                            'samples=14\n'
                                            '   iops        : min=78344, '
                                            'max=79048, avg=78670.43, '
                                            'stdev=204.63, samples=14\n'
                                            '  lat (nsec)   : 1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '10=97.19%, 20=2.74%, 50=0.04%, '
                                            '100=0.02%\n'
                                            '  cpu          : usr=16.16%, '
                                            'sys=44.10%, ctx=550395, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=550525,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=38.4MiB/s (40.3MB/s), '
                                            '38.4MiB/s-38.4MiB/s '
                                            '(40.3MB/s-40.3MB/s), io=269MiB '
                                            '(282MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=790291/0, '
                                            'merge=0/0, ticks=5056/0, '
                                            'in_queue=5056, util=99.06%\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.output'),
                                  'state': {'begin': 1666744177.169167,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.781212568283081,
                                            'end': 1666744177.9503796,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_72.state')},
                       'cmd_73': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.output'),
                                  'state': {'begin': 1666744179.963178,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7831017971038818,
                                            'end': 1666744180.7462797,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_73.state')},
                       'cmd_74': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.output'),
                                  'state': {'begin': 1666744180.7478044,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_aio" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_aio_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.523125410079956,
                                            'end': 1666744192.2709298,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_74.state')},
                       'cmd_75': {'output': 'libaio-bdev_aio: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751281,\n'
                                            '  "timestamp_ms" : '
                                            '1666751281368,\n'
                                            '  "time" : "Wed Oct 26 04:28:01 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_aio",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_aio",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 568576000,\n'
                                            '        "io_kbytes" : 555250,\n'
                                            '        "bw_bytes" : 81225142,\n'
                                            '        "bw" : 79321,\n'
                                            '        "iops" : 158642.857143,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 1110500,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1613,\n'
                                            '          "max" : 37350,\n'
                                            '          "mean" : 1678.867968,\n'
                                            '          "stddev" : 146.532978,\n'
                                            '          "N" : 1110500\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 190,\n'
                                            '          "max" : 115837,\n'
                                            '          "mean" : 4397.280773,\n'
                                            '          "stddev" : '
                                            '1142.935701,\n'
                                            '          "N" : 1110500,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4128,\n'
                                            '            "5.000000" : 4128,\n'
                                            '            "10.000000" : 4128,\n'
                                            '            "20.000000" : 4192,\n'
                                            '            "30.000000" : 4192,\n'
                                            '            "40.000000" : 4192,\n'
                                            '            "50.000000" : 4256,\n'
                                            '            "60.000000" : 4256,\n'
                                            '            "70.000000" : 4320,\n'
                                            '            "80.000000" : 4320,\n'
                                            '            "90.000000" : 4384,\n'
                                            '            "95.000000" : 4576,\n'
                                            '            "99.000000" : 9792,\n'
                                            '            "99.500000" : 11328,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15680,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5730,\n'
                                            '          "max" : 117510,\n'
                                            '          "mean" : 6076.148740,\n'
                                            '          "stddev" : '
                                            '1154.189344,\n'
                                            '          "N" : 1110500\n'
                                            '        },\n'
                                            '        "bw_min" : 79120,\n'
                                            '        "bw_max" : 79623,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '79328.214286,\n'
                                            '        "bw_dev" : 158.211241,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 158240,\n'
                                            '        "iops_max" : 159246,\n'
                                            '        "iops_mean" : '
                                            '158656.500000,\n'
                                            '        "iops_stddev" : '
                                            '316.477913,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 76.614286,\n'
                                            '      "sys_cpu" : 23.357143,\n'
                                            '      "ctx" : 26,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 99.038451,\n'
                                            '        "20" : 0.941108,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.012517,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_aio: (groupid=0, '
                                            'jobs=1): err= 0: pid=139713: Wed '
                                            'Oct 26 04:28:01 2022\n'
                                            '  read: IOPS=159k, BW=77.5MiB/s '
                                            '(81.2MB/s)(542MiB/7000msec)\n'
                                            '    slat (nsec): min=1613, '
                                            'max=37350, avg=1678.87, '
                                            'stdev=146.53\n'
                                            '    clat (nsec): min=190, '
                                            'max=115837, avg=4397.28, '
                                            'stdev=1142.94\n'
                                            '     lat (usec): min=5, max=117, '
                                            'avg= 6.08, stdev= 1.15\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4128],  5.00th=[ '
                                            '4128], 10.00th=[ 4128], 20.00th=[ '
                                            '4192],\n'
                                            '     | 30.00th=[ 4192], 40.00th=[ '
                                            '4192], 50.00th=[ 4256], 60.00th=[ '
                                            '4256],\n'
                                            '     | 70.00th=[ 4320], 80.00th=[ '
                                            '4320], 90.00th=[ 4384], 95.00th=[ '
                                            '4576],\n'
                                            '     | 99.00th=[ 9792], '
                                            '99.50th=[11328], 99.90th=[15552], '
                                            '99.95th=[15680],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=79120, '
                                            'max=79623, per=100.00%, '
                                            'avg=79328.21, stdev=158.21, '
                                            'samples=14\n'
                                            '   iops        : min=158240, '
                                            'max=159246, avg=158656.50, '
                                            'stdev=316.48, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=99.04%, 20=0.94%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=76.61%, '
                                            'sys=23.36%, ctx=26, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1110500,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=77.5MiB/s (81.2MB/s), '
                                            '77.5MiB/s-77.5MiB/s '
                                            '(81.2MB/s-81.2MB/s), io=542MiB '
                                            '(569MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.output'),
                                  'state': {'begin': 1666744192.2780962,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7858402729034424,
                                            'end': 1666744193.0639365,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_75.state')},
                       'cmd_76': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.output'),
                                  'state': {'begin': 1666744195.081775,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7862448692321777,
                                            'end': 1666744195.8680198,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_76.state')},
                       'cmd_77': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.output'),
                                  'state': {'begin': 1666744195.8700564,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_aio" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_aio_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.520851612091064,
                                            'end': 1666744207.390908,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_77.state')},
                       'cmd_78': {'output': 'libaio-bdev_aio: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751296,\n'
                                            '  "timestamp_ms" : '
                                            '1666751296479,\n'
                                            '  "time" : "Wed Oct 26 04:28:16 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_aio",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_aio",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 569852416,\n'
                                            '        "io_kbytes" : 556496,\n'
                                            '        "bw_bytes" : 81407488,\n'
                                            '        "bw" : 79499,\n'
                                            '        "iops" : 158999.000000,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 1112993,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1623,\n'
                                            '          "max" : 25918,\n'
                                            '          "mean" : 1684.411642,\n'
                                            '          "stddev" : 128.915050,\n'
                                            '          "N" : 1112993\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 210,\n'
                                            '          "max" : 76463,\n'
                                            '          "mean" : 4380.684294,\n'
                                            '          "stddev" : '
                                            '1145.616557,\n'
                                            '          "N" : 1112993,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4128,\n'
                                            '            "5.000000" : 4128,\n'
                                            '            "10.000000" : 4128,\n'
                                            '            "20.000000" : 4128,\n'
                                            '            "30.000000" : 4192,\n'
                                            '            "40.000000" : 4192,\n'
                                            '            "50.000000" : 4256,\n'
                                            '            "60.000000" : 4256,\n'
                                            '            "70.000000" : 4256,\n'
                                            '            "80.000000" : 4320,\n'
                                            '            "90.000000" : 4384,\n'
                                            '            "95.000000" : 4512,\n'
                                            '            "99.000000" : 9792,\n'
                                            '            "99.500000" : 11200,\n'
                                            '            "99.900000" : 15424,\n'
                                            '            "99.950000" : 15680,\n'
                                            '            "99.990000" : 57600\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5731,\n'
                                            '          "max" : 78166,\n'
                                            '          "mean" : 6065.095936,\n'
                                            '          "stddev" : '
                                            '1154.036926,\n'
                                            '          "N" : 1112993\n'
                                            '        },\n'
                                            '        "bw_min" : 79322,\n'
                                            '        "bw_max" : 79715,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '79509.214286,\n'
                                            '        "bw_dev" : 119.767196,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 158644,\n'
                                            '        "iops_max" : 159430,\n'
                                            '        "iops_mean" : '
                                            '159018.571429,\n'
                                            '        "iops_stddev" : '
                                            '239.596364,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 76.442857,\n'
                                            '      "sys_cpu" : 23.528571,\n'
                                            '      "ctx" : 26,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 99.035214,\n'
                                            '        "20" : 0.943762,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013208,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_aio: (groupid=0, '
                                            'jobs=1): err= 0: pid=139762: Wed '
                                            'Oct 26 04:28:16 2022\n'
                                            '  read: IOPS=159k, BW=77.6MiB/s '
                                            '(81.4MB/s)(543MiB/7000msec)\n'
                                            '    slat (nsec): min=1623, '
                                            'max=25918, avg=1684.41, '
                                            'stdev=128.92\n'
                                            '    clat (nsec): min=210, '
                                            'max=76463, avg=4380.68, '
                                            'stdev=1145.62\n'
                                            '     lat (nsec): min=5731, '
                                            'max=78166, avg=6065.10, '
                                            'stdev=1154.04\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4128],  5.00th=[ '
                                            '4128], 10.00th=[ 4128], 20.00th=[ '
                                            '4128],\n'
                                            '     | 30.00th=[ 4192], 40.00th=[ '
                                            '4192], 50.00th=[ 4256], 60.00th=[ '
                                            '4256],\n'
                                            '     | 70.00th=[ 4256], 80.00th=[ '
                                            '4320], 90.00th=[ 4384], 95.00th=[ '
                                            '4512],\n'
                                            '     | 99.00th=[ 9792], '
                                            '99.50th=[11200], 99.90th=[15424], '
                                            '99.95th=[15680],\n'
                                            '     | 99.99th=[57600]\n'
                                            '   bw (  KiB/s): min=79322, '
                                            'max=79715, per=100.00%, '
                                            'avg=79509.21, stdev=119.77, '
                                            'samples=14\n'
                                            '   iops        : min=158644, '
                                            'max=159430, avg=159018.57, '
                                            'stdev=239.60, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=99.04%, 20=0.94%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=76.44%, '
                                            'sys=23.53%, ctx=26, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1112993,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=77.6MiB/s (81.4MB/s), '
                                            '77.6MiB/s-77.6MiB/s '
                                            '(81.4MB/s-81.4MB/s), io=543MiB '
                                            '(570MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.output'),
                                  'state': {'begin': 1666744207.3945053,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8608551025390625,
                                            'end': 1666744208.2553604,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_78.state')},
                       'cmd_79': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.output'),
                                  'state': {'begin': 1666744210.2829776,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7859225273132324,
                                            'end': 1666744211.0689,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_79.state')},
                       'cmd_80': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.output'),
                                  'state': {'begin': 1666744211.0708146,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_aio" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_aio_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.525362253189087,
                                            'end': 1666744222.5961769,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_80.state')},
                       'cmd_81': {'output': 'libaio-bdev_aio: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751311,\n'
                                            '  "timestamp_ms" : '
                                            '1666751311692,\n'
                                            '  "time" : "Wed Oct 26 04:28:31 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_aio",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_aio",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 570132480,\n'
                                            '        "io_kbytes" : 556770,\n'
                                            '        "bw_bytes" : 81435863,\n'
                                            '        "bw" : 79527,\n'
                                            '        "iops" : 159054.277960,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1113539,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1603,\n'
                                            '          "max" : 60874,\n'
                                            '          "mean" : 1654.208657,\n'
                                            '          "stddev" : 140.431430,\n'
                                            '          "N" : 1113540\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 200,\n'
                                            '          "max" : 114164,\n'
                                            '          "mean" : 4398.236214,\n'
                                            '          "stddev" : '
                                            '1110.230874,\n'
                                            '          "N" : 1113539,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4128,\n'
                                            '            "5.000000" : 4128,\n'
                                            '            "10.000000" : 4192,\n'
                                            '            "20.000000" : 4192,\n'
                                            '            "30.000000" : 4192,\n'
                                            '            "40.000000" : 4192,\n'
                                            '            "50.000000" : 4256,\n'
                                            '            "60.000000" : 4256,\n'
                                            '            "70.000000" : 4320,\n'
                                            '            "80.000000" : 4320,\n'
                                            '            "90.000000" : 4384,\n'
                                            '            "95.000000" : 4512,\n'
                                            '            "99.000000" : 9920,\n'
                                            '            "99.500000" : 10560,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15680,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5730,\n'
                                            '          "max" : 115827,\n'
                                            '          "mean" : 6052.444891,\n'
                                            '          "stddev" : '
                                            '1120.267316,\n'
                                            '          "N" : 1113539\n'
                                            '        },\n'
                                            '        "bw_min" : 79426,\n'
                                            '        "bw_max" : 79846,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '79546.428571,\n'
                                            '        "bw_dev" : 125.765175,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 158852,\n'
                                            '        "iops_max" : 159693,\n'
                                            '        "iops_mean" : '
                                            '159092.785714,\n'
                                            '        "iops_stddev" : '
                                            '251.830889,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 76.100000,\n'
                                            '      "sys_cpu" : 23.871429,\n'
                                            '      "ctx" : 25,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 99.023653,\n'
                                            '        "20" : 0.955692,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.011674,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_aio: (groupid=0, '
                                            'jobs=1): err= 0: pid=139811: Wed '
                                            'Oct 26 04:28:31 2022\n'
                                            '  read: IOPS=159k, BW=77.7MiB/s '
                                            '(81.4MB/s)(544MiB/7001msec)\n'
                                            '    slat (nsec): min=1603, '
                                            'max=60874, avg=1654.21, '
                                            'stdev=140.43\n'
                                            '    clat (nsec): min=200, '
                                            'max=114164, avg=4398.24, '
                                            'stdev=1110.23\n'
                                            '     lat (usec): min=5, max=115, '
                                            'avg= 6.05, stdev= 1.12\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4128],  5.00th=[ '
                                            '4128], 10.00th=[ 4192], 20.00th=[ '
                                            '4192],\n'
                                            '     | 30.00th=[ 4192], 40.00th=[ '
                                            '4192], 50.00th=[ 4256], 60.00th=[ '
                                            '4256],\n'
                                            '     | 70.00th=[ 4320], 80.00th=[ '
                                            '4320], 90.00th=[ 4384], 95.00th=[ '
                                            '4512],\n'
                                            '     | 99.00th=[ 9920], '
                                            '99.50th=[10560], 99.90th=[15552], '
                                            '99.95th=[15680],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=79426, '
                                            'max=79846, per=100.00%, '
                                            'avg=79546.43, stdev=125.77, '
                                            'samples=14\n'
                                            '   iops        : min=158852, '
                                            'max=159693, avg=159092.79, '
                                            'stdev=251.83, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=99.02%, 20=0.96%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=76.10%, '
                                            'sys=23.87%, ctx=25, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1113539,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=77.7MiB/s (81.4MB/s), '
                                            '77.7MiB/s-77.7MiB/s '
                                            '(81.4MB/s-81.4MB/s), io=544MiB '
                                            '(570MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.output'),
                                  'state': {'begin': 1666744222.599019,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_aio_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7893621921539307,
                                            'end': 1666744223.3883812,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_81.state')},
                       'cmd_82': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.output'),
                                  'state': {'begin': 1666744225.4102664,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7753422260284424,
                                            'end': 1666744226.1856086,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_82.state')},
                       'cmd_83': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.output'),
                                  'state': {'begin': 1666744226.187541,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.548415660858154,
                                            'end': 1666744237.7359567,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_83.state')},
                       'cmd_84': {'output': 'libaio-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751326,\n'
                                            '  "timestamp_ms" : '
                                            '1666751326817,\n'
                                            '  "time" : "Wed Oct 26 04:28:46 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 549307392,\n'
                                            '        "io_kbytes" : 536433,\n'
                                            '        "bw_bytes" : 78461275,\n'
                                            '        "bw" : 76622,\n'
                                            '        "iops" : 153244.679332,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1072866,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1623,\n'
                                            '          "max" : 14217,\n'
                                            '          "mean" : 1678.038774,\n'
                                            '          "stddev" : 119.906182,\n'
                                            '          "N" : 1072866\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 591,\n'
                                            '          "max" : 118713,\n'
                                            '          "mean" : 4583.107354,\n'
                                            '          "stddev" : '
                                            '1160.723079,\n'
                                            '          "N" : 1072866,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4320,\n'
                                            '            "5.000000" : 4320,\n'
                                            '            "10.000000" : 4320,\n'
                                            '            "20.000000" : 4320,\n'
                                            '            "30.000000" : 4320,\n'
                                            '            "40.000000" : 4320,\n'
                                            '            "50.000000" : 4384,\n'
                                            '            "60.000000" : 4384,\n'
                                            '            "70.000000" : 4640,\n'
                                            '            "80.000000" : 4640,\n'
                                            '            "90.000000" : 4704,\n'
                                            '            "95.000000" : 4768,\n'
                                            '            "99.000000" : 9792,\n'
                                            '            "99.500000" : 10816,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15680,\n'
                                            '            "99.990000" : 59136\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5881,\n'
                                            '          "max" : 120406,\n'
                                            '          "mean" : 6261.146127,\n'
                                            '          "stddev" : '
                                            '1168.029557,\n'
                                            '          "N" : 1072866\n'
                                            '        },\n'
                                            '        "bw_min" : 76498,\n'
                                            '        "bw_max" : 76736,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '76644.785714,\n'
                                            '        "bw_dev" : 66.817983,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 152997,\n'
                                            '        "iops_max" : 153472,\n'
                                            '        "iops_mean" : '
                                            '153289.642857,\n'
                                            '        "iops_stddev" : '
                                            '133.467142,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 36.614286,\n'
                                            '      "sys_cpu" : 63.357143,\n'
                                            '      "ctx" : 23,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 99.073137,\n'
                                            '        "20" : 0.903188,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.015379,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=139861: Wed '
                                            'Oct 26 04:28:46 2022\n'
                                            '  read: IOPS=153k, BW=74.8MiB/s '
                                            '(78.5MB/s)(524MiB/7001msec)\n'
                                            '    slat (nsec): min=1623, '
                                            'max=14217, avg=1678.04, '
                                            'stdev=119.91\n'
                                            '    clat (nsec): min=591, '
                                            'max=118713, avg=4583.11, '
                                            'stdev=1160.72\n'
                                            '     lat (usec): min=5, max=120, '
                                            'avg= 6.26, stdev= 1.17\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4320],  5.00th=[ '
                                            '4320], 10.00th=[ 4320], 20.00th=[ '
                                            '4320],\n'
                                            '     | 30.00th=[ 4320], 40.00th=[ '
                                            '4320], 50.00th=[ 4384], 60.00th=[ '
                                            '4384],\n'
                                            '     | 70.00th=[ 4640], 80.00th=[ '
                                            '4640], 90.00th=[ 4704], 95.00th=[ '
                                            '4768],\n'
                                            '     | 99.00th=[ 9792], '
                                            '99.50th=[10816], 99.90th=[15552], '
                                            '99.95th=[15680],\n'
                                            '     | 99.99th=[59136]\n'
                                            '   bw (  KiB/s): min=76498, '
                                            'max=76736, per=100.00%, '
                                            'avg=76644.79, stdev=66.82, '
                                            'samples=14\n'
                                            '   iops        : min=152997, '
                                            'max=153472, avg=153289.64, '
                                            'stdev=133.47, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=99.07%, 20=0.90%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.02%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=36.61%, '
                                            'sys=63.36%, ctx=23, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1072866,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=74.8MiB/s (78.5MB/s), '
                                            '74.8MiB/s-74.8MiB/s '
                                            '(78.5MB/s-78.5MB/s), io=524MiB '
                                            '(549MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.output'),
                                  'state': {'begin': 1666744237.7385774,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7844376564025879,
                                            'end': 1666744238.523015,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_84.state')},
                       'cmd_85': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.output'),
                                  'state': {'begin': 1666744240.5375273,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7823998928070068,
                                            'end': 1666744241.3199272,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_85.state')},
                       'cmd_86': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.output'),
                                  'state': {'begin': 1666744241.3217847,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.508594512939453,
                                            'end': 1666744252.8303792,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_86.state')},
                       'cmd_87': {'output': 'libaio-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751341,\n'
                                            '  "timestamp_ms" : '
                                            '1666751341933,\n'
                                            '  "time" : "Wed Oct 26 04:29:01 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 548251136,\n'
                                            '        "io_kbytes" : 535401,\n'
                                            '        "bw_bytes" : 78310403,\n'
                                            '        "bw" : 76475,\n'
                                            '        "iops" : 152950.007142,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1070803,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1633,\n'
                                            '          "max" : 14327,\n'
                                            '          "mean" : 1697.999735,\n'
                                            '          "stddev" : 113.979161,\n'
                                            '          "N" : 1070803\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 602,\n'
                                            '          "max" : 97693,\n'
                                            '          "mean" : 4606.212420,\n'
                                            '          "stddev" : '
                                            '1101.921202,\n'
                                            '          "N" : 1070803,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4384,\n'
                                            '            "5.000000" : 4384,\n'
                                            '            "10.000000" : 4384,\n'
                                            '            "20.000000" : 4384,\n'
                                            '            "30.000000" : 4384,\n'
                                            '            "40.000000" : 4448,\n'
                                            '            "50.000000" : 4448,\n'
                                            '            "60.000000" : 4448,\n'
                                            '            "70.000000" : 4448,\n'
                                            '            "80.000000" : 4512,\n'
                                            '            "90.000000" : 4704,\n'
                                            '            "95.000000" : 4768,\n'
                                            '            "99.000000" : 9792,\n'
                                            '            "99.500000" : 10816,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15680,\n'
                                            '            "99.990000" : 57088\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5921,\n'
                                            '          "max" : 99376,\n'
                                            '          "mean" : 6304.212155,\n'
                                            '          "stddev" : '
                                            '1109.513640,\n'
                                            '          "N" : 1070803\n'
                                            '        },\n'
                                            '        "bw_min" : 76350,\n'
                                            '        "bw_max" : 76686,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '76496.357143,\n'
                                            '        "bw_dev" : 105.615855,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 152700,\n'
                                            '        "iops_max" : 153372,\n'
                                            '        "iops_mean" : '
                                            '152992.928571,\n'
                                            '        "iops_stddev" : '
                                            '211.019673,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 29.985714,\n'
                                            '      "sys_cpu" : 69.985714,\n'
                                            '      "ctx" : 20,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 99.040907,\n'
                                            '        "20" : 0.937054,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.013168,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=139913: Wed '
                                            'Oct 26 04:29:01 2022\n'
                                            '  read: IOPS=153k, BW=74.7MiB/s '
                                            '(78.3MB/s)(523MiB/7001msec)\n'
                                            '    slat (nsec): min=1633, '
                                            'max=14327, avg=1698.00, '
                                            'stdev=113.98\n'
                                            '    clat (nsec): min=602, '
                                            'max=97693, avg=4606.21, '
                                            'stdev=1101.92\n'
                                            '     lat (nsec): min=5921, '
                                            'max=99376, avg=6304.21, '
                                            'stdev=1109.51\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4384],  5.00th=[ '
                                            '4384], 10.00th=[ 4384], 20.00th=[ '
                                            '4384],\n'
                                            '     | 30.00th=[ 4384], 40.00th=[ '
                                            '4448], 50.00th=[ 4448], 60.00th=[ '
                                            '4448],\n'
                                            '     | 70.00th=[ 4448], 80.00th=[ '
                                            '4512], 90.00th=[ 4704], 95.00th=[ '
                                            '4768],\n'
                                            '     | 99.00th=[ 9792], '
                                            '99.50th=[10816], 99.90th=[15552], '
                                            '99.95th=[15680],\n'
                                            '     | 99.99th=[57088]\n'
                                            '   bw (  KiB/s): min=76350, '
                                            'max=76686, per=100.00%, '
                                            'avg=76496.36, stdev=105.62, '
                                            'samples=14\n'
                                            '   iops        : min=152700, '
                                            'max=153372, avg=152992.93, '
                                            'stdev=211.02, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=99.04%, 20=0.94%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=29.99%, '
                                            'sys=69.99%, ctx=20, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1070803,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=74.7MiB/s (78.3MB/s), '
                                            '74.7MiB/s-74.7MiB/s '
                                            '(78.3MB/s-78.3MB/s), io=523MiB '
                                            '(548MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.output'),
                                  'state': {'begin': 1666744252.833364,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7898769378662109,
                                            'end': 1666744253.623241,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_87.state')},
                       'cmd_88': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.output'),
                                  'state': {'begin': 1666744255.6336243,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7826905250549316,
                                            'end': 1666744256.4163148,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_88.state')},
                       'cmd_89': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.output'),
                                  'state': {'begin': 1666744256.4187577,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.516239881515503,
                                            'end': 1666744267.9349976,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_89.state')},
                       'cmd_90': {'output': 'libaio-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 512B-512B, '
                                            '(W) 512B-512B, (T) 512B-512B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751357,\n'
                                            '  "timestamp_ms" : '
                                            '1666751357029,\n'
                                            '  "time" : "Wed Oct 26 04:29:17 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                            '        "filename" : "Nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 552497664,\n'
                                            '        "io_kbytes" : 539548,\n'
                                            '        "bw_bytes" : 78916963,\n'
                                            '        "bw" : 77067,\n'
                                            '        "iops" : 154134.695044,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 1079097,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1613,\n'
                                            '          "max" : 25488,\n'
                                            '          "mean" : 1684.209908,\n'
                                            '          "stddev" : 120.441497,\n'
                                            '          "N" : 1079097\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 592,\n'
                                            '          "max" : 76343,\n'
                                            '          "mean" : 4568.302466,\n'
                                            '          "stddev" : '
                                            '1090.399727,\n'
                                            '          "N" : 1079097,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 4320,\n'
                                            '            "5.000000" : 4320,\n'
                                            '            "10.000000" : 4320,\n'
                                            '            "20.000000" : 4384,\n'
                                            '            "30.000000" : 4384,\n'
                                            '            "40.000000" : 4384,\n'
                                            '            "50.000000" : 4384,\n'
                                            '            "60.000000" : 4448,\n'
                                            '            "70.000000" : 4448,\n'
                                            '            "80.000000" : 4448,\n'
                                            '            "90.000000" : 4704,\n'
                                            '            "95.000000" : 4768,\n'
                                            '            "99.000000" : 9792,\n'
                                            '            "99.500000" : 11328,\n'
                                            '            "99.900000" : 15552,\n'
                                            '            "99.950000" : 15680,\n'
                                            '            "99.990000" : 56576\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 5861,\n'
                                            '          "max" : 78016,\n'
                                            '          "mean" : 6252.512374,\n'
                                            '          "stddev" : '
                                            '1098.558092,\n'
                                            '          "N" : 1079097\n'
                                            '        },\n'
                                            '        "bw_min" : 76780,\n'
                                            '        "bw_max" : 77307,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '77088.357143,\n'
                                            '        "bw_dev" : 198.919236,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 153560,\n'
                                            '        "iops_max" : 154614,\n'
                                            '        "iops_mean" : '
                                            '154176.714286,\n'
                                            '        "iops_stddev" : '
                                            '397.838473,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 29.242857,\n'
                                            '      "sys_cpu" : 70.728571,\n'
                                            '      "ctx" : 22,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 99.054580,\n'
                                            '        "20" : 0.927349,\n'
                                            '        "50" : 0.010000,\n'
                                            '        "100" : 0.010842,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=139972: Wed '
                                            'Oct 26 04:29:17 2022\n'
                                            '  read: IOPS=154k, BW=75.3MiB/s '
                                            '(78.9MB/s)(527MiB/7001msec)\n'
                                            '    slat (nsec): min=1613, '
                                            'max=25488, avg=1684.21, '
                                            'stdev=120.44\n'
                                            '    clat (nsec): min=592, '
                                            'max=76343, avg=4568.30, '
                                            'stdev=1090.40\n'
                                            '     lat (nsec): min=5861, '
                                            'max=78016, avg=6252.51, '
                                            'stdev=1098.56\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 4320],  5.00th=[ '
                                            '4320], 10.00th=[ 4320], 20.00th=[ '
                                            '4384],\n'
                                            '     | 30.00th=[ 4384], 40.00th=[ '
                                            '4384], 50.00th=[ 4384], 60.00th=[ '
                                            '4448],\n'
                                            '     | 70.00th=[ 4448], 80.00th=[ '
                                            '4448], 90.00th=[ 4704], 95.00th=[ '
                                            '4768],\n'
                                            '     | 99.00th=[ 9792], '
                                            '99.50th=[11328], 99.90th=[15552], '
                                            '99.95th=[15680],\n'
                                            '     | 99.99th=[56576]\n'
                                            '   bw (  KiB/s): min=76780, '
                                            'max=77307, per=100.00%, '
                                            'avg=77088.36, stdev=198.92, '
                                            'samples=14\n'
                                            '   iops        : min=153560, '
                                            'max=154614, avg=154176.71, '
                                            'stdev=397.84, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%, '
                                            '1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=99.05%, 20=0.93%, '
                                            '50=0.01%\n'
                                            '  lat (usec)   : 100=0.01%\n'
                                            '  cpu          : usr=29.24%, '
                                            'sys=70.73%, ctx=22, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=1079097,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=75.3MiB/s (78.9MB/s), '
                                            '75.3MiB/s-75.3MiB/s '
                                            '(78.9MB/s-78.9MB/s), io=527MiB '
                                            '(552MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.output'),
                                  'state': {'begin': 1666744267.9379344,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7892000675201416,
                                            'end': 1666744268.7271345,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_90.state')},
                       'cmd_91': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.output'),
                                  'state': {'begin': 1666744269.753349,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7843270301818848,
                                            'end': 1666744270.537676,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_91.state')},
                       'cmd_92': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.output'),
                                  'state': {'begin': 1666744270.5395074,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="libaio" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-xnvme_0.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.08480453491211,
                                            'end': 1666744281.624312,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_92.state')},
                       'cmd_93': {'output': 'libaio-xnvme: (g=0): rw=randread, '
                                            'bs=(R) 512B-512B, (W) 512B-512B, '
                                            '(T) 512B-512B, ioengine=xnvme, '
                                            'iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751370,\n'
                                            '  "timestamp_ms" : '
                                            '1666751370821,\n'
                                            '  "time" : "Wed Oct 26 04:29:30 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : "libaio-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"libaio",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 263074816,\n'
                                            '        "io_kbytes" : 256909,\n'
                                            '        "bw_bytes" : 37576748,\n'
                                            '        "bw" : 36696,\n'
                                            '        "iops" : 73392.086845,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 513818,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2856,\n'
                                            '          "max" : 24596,\n'
                                            '          "mean" : 3174.835060,\n'
                                            '          "stddev" : 192.695012,\n'
                                            '          "N" : 513818\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 942,\n'
                                            '          "max" : 70743,\n'
                                            '          "mean" : 9889.292977,\n'
                                            '          "stddev" : '
                                            '1172.701180,\n'
                                            '          "N" : 513818,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8512,\n'
                                            '            "5.000000" : 9536,\n'
                                            '            "10.000000" : 9664,\n'
                                            '            "20.000000" : 9664,\n'
                                            '            "30.000000" : 9792,\n'
                                            '            "40.000000" : 9792,\n'
                                            '            "50.000000" : 9792,\n'
                                            '            "60.000000" : 9792,\n'
                                            '            "70.000000" : 9792,\n'
                                            '            "80.000000" : 9792,\n'
                                            '            "90.000000" : 9920,\n'
                                            '            "95.000000" : 10048,\n'
                                            '            "99.000000" : 15040,\n'
                                            '            "99.500000" : 16192,\n'
                                            '            "99.900000" : 20096,\n'
                                            '            "99.950000" : 24960,\n'
                                            '            "99.990000" : 58112\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 8586,\n'
                                            '          "max" : 73798,\n'
                                            '          "mean" : 13064.128038,\n'
                                            '          "stddev" : '
                                            '1183.145219,\n'
                                            '          "N" : 513818\n'
                                            '        },\n'
                                            '        "bw_min" : 36599,\n'
                                            '        "bw_max" : 36839,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '36707.785714,\n'
                                            '        "bw_dev" : 78.708985,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 73198,\n'
                                            '        "iops_max" : 73678,\n'
                                            '        "iops_mean" : '
                                            '73415.428571,\n'
                                            '        "iops_stddev" : '
                                            '157.295271,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 15.328571,\n'
                                            '      "sys_cpu" : 47.942857,\n'
                                            '      "ctx" : 513642,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010120,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 94.480731,\n'
                                            '        "20" : 5.399772,\n'
                                            '        "50" : 0.087969,\n'
                                            '        "100" : 0.019657,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=140012: Wed '
                                            'Oct 26 04:29:30 2022\n'
                                            '  read: IOPS=73.4k, BW=35.8MiB/s '
                                            '(37.6MB/s)(251MiB/7001msec)\n'
                                            '    slat (nsec): min=2856, '
                                            'max=24596, avg=3174.84, '
                                            'stdev=192.70\n'
                                            '    clat (nsec): min=942, '
                                            'max=70743, avg=9889.29, '
                                            'stdev=1172.70\n'
                                            '     lat (nsec): min=8586, '
                                            'max=73798, avg=13064.13, '
                                            'stdev=1183.15\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8512],  5.00th=[ '
                                            '9536], 10.00th=[ 9664], 20.00th=[ '
                                            '9664],\n'
                                            '     | 30.00th=[ 9792], 40.00th=[ '
                                            '9792], 50.00th=[ 9792], 60.00th=[ '
                                            '9792],\n'
                                            '     | 70.00th=[ 9792], 80.00th=[ '
                                            '9792], 90.00th=[ 9920], '
                                            '95.00th=[10048],\n'
                                            '     | 99.00th=[15040], '
                                            '99.50th=[16192], 99.90th=[20096], '
                                            '99.95th=[24960],\n'
                                            '     | 99.99th=[58112]\n'
                                            '   bw (  KiB/s): min=36599, '
                                            'max=36839, per=100.00%, '
                                            'avg=36707.79, stdev=78.71, '
                                            'samples=14\n'
                                            '   iops        : min=73198, '
                                            'max=73678, avg=73415.43, '
                                            'stdev=157.30, samples=14\n'
                                            '  lat (nsec)   : 1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '10=94.48%, 20=5.40%, 50=0.09%, '
                                            '100=0.02%\n'
                                            '  cpu          : usr=15.33%, '
                                            'sys=47.94%, ctx=513642, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=513818,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=35.8MiB/s (37.6MB/s), '
                                            '35.8MiB/s-35.8MiB/s '
                                            '(37.6MB/s-37.6MB/s), io=251MiB '
                                            '(263MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.output'),
                                  'state': {'begin': 1666744281.6314862,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7852046489715576,
                                            'end': 1666744282.4166908,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_93.state')},
                       'cmd_94': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.output'),
                                  'state': {'begin': 1666744283.4084666,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7857820987701416,
                                            'end': 1666744284.1942487,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_94.state')},
                       'cmd_95': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.output'),
                                  'state': {'begin': 1666744284.1963353,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="libaio" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-xnvme_1.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.09076476097107,
                                            'end': 1666744295.2871,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_95.state')},
                       'cmd_96': {'output': 'libaio-xnvme: (g=0): rw=randread, '
                                            'bs=(R) 512B-512B, (W) 512B-512B, '
                                            '(T) 512B-512B, ioengine=xnvme, '
                                            'iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751384,\n'
                                            '  "timestamp_ms" : '
                                            '1666751384484,\n'
                                            '  "time" : "Wed Oct 26 04:29:44 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : "libaio-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"libaio",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 345976320,\n'
                                            '        "io_kbytes" : 337867,\n'
                                            '        "bw_bytes" : 49418128,\n'
                                            '        "bw" : 48259,\n'
                                            '        "iops" : 96519.782888,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 675735,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2174,\n'
                                            '          "max" : 24436,\n'
                                            '          "mean" : 2431.272401,\n'
                                            '          "stddev" : 151.964015,\n'
                                            '          "N" : 675735\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 701,\n'
                                            '          "max" : 75923,\n'
                                            '          "mean" : 7477.687172,\n'
                                            '          "stddev" : '
                                            '1131.118569,\n'
                                            '          "N" : 675735,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 7136,\n'
                                            '            "5.000000" : 7200,\n'
                                            '            "10.000000" : 7200,\n'
                                            '            "20.000000" : 7264,\n'
                                            '            "30.000000" : 7264,\n'
                                            '            "40.000000" : 7264,\n'
                                            '            "50.000000" : 7328,\n'
                                            '            "60.000000" : 7328,\n'
                                            '            "70.000000" : 7392,\n'
                                            '            "80.000000" : 7392,\n'
                                            '            "90.000000" : 7584,\n'
                                            '            "95.000000" : 7712,\n'
                                            '            "99.000000" : 14016,\n'
                                            '            "99.500000" : 15040,\n'
                                            '            "99.900000" : 16064,\n'
                                            '            "99.950000" : 17024,\n'
                                            '            "99.990000" : 56064\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 7504,\n'
                                            '          "max" : 78317,\n'
                                            '          "mean" : 9908.959573,\n'
                                            '          "stddev" : '
                                            '1141.464837,\n'
                                            '          "N" : 675735\n'
                                            '        },\n'
                                            '        "bw_min" : 48137,\n'
                                            '        "bw_max" : 48485,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '48272.857143,\n'
                                            '        "bw_dev" : 96.448358,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 96274,\n'
                                            '        "iops_max" : 96970,\n'
                                            '        "iops_mean" : '
                                            '96545.785714,\n'
                                            '        "iops_stddev" : '
                                            '192.890634,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 16.128571,\n'
                                            '      "sys_cpu" : 46.542857,\n'
                                            '      "ctx" : 675583,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 98.266776,\n'
                                            '        "20" : 1.699483,\n'
                                            '        "50" : 0.011839,\n'
                                            '        "100" : 0.013023,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=140052: Wed '
                                            'Oct 26 04:29:44 2022\n'
                                            '  read: IOPS=96.5k, BW=47.1MiB/s '
                                            '(49.4MB/s)(330MiB/7001msec)\n'
                                            '    slat (nsec): min=2174, '
                                            'max=24436, avg=2431.27, '
                                            'stdev=151.96\n'
                                            '    clat (nsec): min=701, '
                                            'max=75923, avg=7477.69, '
                                            'stdev=1131.12\n'
                                            '     lat (nsec): min=7504, '
                                            'max=78317, avg=9908.96, '
                                            'stdev=1141.46\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 7136],  5.00th=[ '
                                            '7200], 10.00th=[ 7200], 20.00th=[ '
                                            '7264],\n'
                                            '     | 30.00th=[ 7264], 40.00th=[ '
                                            '7264], 50.00th=[ 7328], 60.00th=[ '
                                            '7328],\n'
                                            '     | 70.00th=[ 7392], 80.00th=[ '
                                            '7392], 90.00th=[ 7584], 95.00th=[ '
                                            '7712],\n'
                                            '     | 99.00th=[14016], '
                                            '99.50th=[15040], 99.90th=[16064], '
                                            '99.95th=[17024],\n'
                                            '     | 99.99th=[56064]\n'
                                            '   bw (  KiB/s): min=48137, '
                                            'max=48485, per=100.00%, '
                                            'avg=48272.86, stdev=96.45, '
                                            'samples=14\n'
                                            '   iops        : min=96274, '
                                            'max=96970, avg=96545.79, '
                                            'stdev=192.89, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%, '
                                            '1000=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=98.27%, 20=1.70%, 50=0.01%, '
                                            '100=0.01%\n'
                                            '  cpu          : usr=16.13%, '
                                            'sys=46.54%, ctx=675583, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=675735,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=47.1MiB/s (49.4MB/s), '
                                            '47.1MiB/s-47.1MiB/s '
                                            '(49.4MB/s-49.4MB/s), io=330MiB '
                                            '(346MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.output'),
                                  'state': {'begin': 1666744295.290078,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7871439456939697,
                                            'end': 1666744296.0772219,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_96.state')},
                       'cmd_97': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.output'),
                                  'state': {'begin': 1666744297.0662894,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7880120277404785,
                                            'end': 1666744297.8543015,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_97.state')},
                       'cmd_98': {'output': '',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.output'),
                                  'state': {'begin': 1666744297.8563988,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="512" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="libaio" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_512_1_libaio-xnvme_2.txt" '
                                                   '--cpus_allowed="1"',
                                            'cwd': 'None',
                                            'elapsed': 11.089160203933716,
                                            'end': 1666744308.945559,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_98.state')},
                       'cmd_99': {'output': 'libaio-xnvme: (g=0): rw=randread, '
                                            'bs=(R) 512B-512B, (W) 512B-512B, '
                                            '(T) 512B-512B, ioengine=xnvme, '
                                            'iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666751398,\n'
                                            '  "timestamp_ms" : '
                                            '1666751398140,\n'
                                            '  "time" : "Wed Oct 26 04:29:58 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : "libaio-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "512",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"libaio",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1",\n'
                                            '        "cpus_allowed" : "1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 264449536,\n'
                                            '        "io_kbytes" : 258251,\n'
                                            '        "bw_bytes" : 37773108,\n'
                                            '        "bw" : 36887,\n'
                                            '        "iops" : 73775.460648,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 516502,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2955,\n'
                                            '          "max" : 35957,\n'
                                            '          "mean" : 3190.098706,\n'
                                            '          "stddev" : 227.777047,\n'
                                            '          "N" : 516503\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 892,\n'
                                            '          "max" : 117179,\n'
                                            '          "mean" : 9770.164466,\n'
                                            '          "stddev" : '
                                            '1173.998407,\n'
                                            '          "N" : 516502,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8256,\n'
                                            '            "5.000000" : 9408,\n'
                                            '            "10.000000" : 9536,\n'
                                            '            "20.000000" : 9536,\n'
                                            '            "30.000000" : 9664,\n'
                                            '            "40.000000" : 9664,\n'
                                            '            "50.000000" : 9664,\n'
                                            '            "60.000000" : 9664,\n'
                                            '            "70.000000" : 9664,\n'
                                            '            "80.000000" : 9792,\n'
                                            '            "90.000000" : 9792,\n'
                                            '            "95.000000" : 9920,\n'
                                            '            "99.000000" : 14912,\n'
                                            '            "99.500000" : 16064,\n'
                                            '            "99.900000" : 20096,\n'
                                            '            "99.950000" : 25216,\n'
                                            '            "99.990000" : 55552\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9898,\n'
                                            '          "max" : 120415,\n'
                                            '          "mean" : 12960.263335,\n'
                                            '          "stddev" : '
                                            '1193.805394,\n'
                                            '          "N" : 516502\n'
                                            '        },\n'
                                            '        "bw_min" : 36783,\n'
                                            '        "bw_max" : 37012,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '36898.714286,\n'
                                            '        "bw_dev" : 75.942720,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 73566,\n'
                                            '        "iops_max" : 74024,\n'
                                            '        "iops_mean" : '
                                            '73797.714286,\n'
                                            '        "iops_stddev" : '
                                            '151.591774,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 15.985714,\n'
                                            '      "sys_cpu" : 47.085714,\n'
                                            '      "ctx" : 516350,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 95.703211,\n'
                                            '        "20" : 4.180429,\n'
                                            '        "50" : 0.089061,\n'
                                            '        "100" : 0.017812,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=140092: Wed '
                                            'Oct 26 04:29:58 2022\n'
                                            '  read: IOPS=73.8k, BW=36.0MiB/s '
                                            '(37.8MB/s)(252MiB/7001msec)\n'
                                            '    slat (nsec): min=2955, '
                                            'max=35957, avg=3190.10, '
                                            'stdev=227.78\n'
                                            '    clat (nsec): min=892, '
                                            'max=117179, avg=9770.16, '
                                            'stdev=1174.00\n'
                                            '     lat (usec): min=9, max=120, '
                                            'avg=12.96, stdev= 1.19\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8256],  5.00th=[ '
                                            '9408], 10.00th=[ 9536], 20.00th=[ '
                                            '9536],\n'
                                            '     | 30.00th=[ 9664], 40.00th=[ '
                                            '9664], 50.00th=[ 9664], 60.00th=[ '
                                            '9664],\n'
                                            '     | 70.00th=[ 9664], 80.00th=[ '
                                            '9792], 90.00th=[ 9792], 95.00th=[ '
                                            '9920],\n'
                                            '     | 99.00th=[14912], '
                                            '99.50th=[16064], 99.90th=[20096], '
                                            '99.95th=[25216],\n'
                                            '     | 99.99th=[55552]\n'
                                            '   bw (  KiB/s): min=36783, '
                                            'max=37012, per=100.00%, '
                                            'avg=36898.71, stdev=75.94, '
                                            'samples=14\n'
                                            '   iops        : min=73566, '
                                            'max=74024, avg=73797.71, '
                                            'stdev=151.59, samples=14\n'
                                            '  lat (nsec)   : 1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '10=95.70%, 20=4.18%, 50=0.09%, '
                                            '100=0.02%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=15.99%, '
                                            'sys=47.09%, ctx=516350, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=516502,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=36.0MiB/s (37.8MB/s), '
                                            '36.0MiB/s-36.0MiB/s '
                                            '(37.8MB/s-37.8MB/s), io=252MiB '
                                            '(264MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.output'),
                                  'state': {'begin': 1666744308.9486403,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_512_1_libaio-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7886214256286621,
                                            'end': 1666744309.7372618,
                                            'err': 0,
                                            'output_dpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio',
                                            'output_fpath': '/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.output'},
                                  'state_path': PosixPath('/home/safl/git/spdkvf2022/cijoe-output/003_fio/cmd_99.state')}}},
 'id': '003_fio',
 'name': 'fio',
 'nr': 3,
 'status': {'elapsed': 1891.1022799015045,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'fioe',
 'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['512'], 'repetitions': 3}}
$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666750939,
  "timestamp_ms" : 1666750939831,
  "time" : "Wed Oct 26 04:22:19 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 298279936,
        "io_kbytes" : 291289,
        "bw_bytes" : 42605332,
        "bw" : 41606,
        "iops" : 83213.540923,
        "runtime" : 7001,
        "total_ios" : 582578,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2224,
          "max" : 28664,
          "mean" : 2408.062177,
          "stddev" : 216.254296,
          "N" : 582578
        },
        "clat_ns" : {
          "min" : 171,
          "max" : 108203,
          "mean" : 9002.665866,
          "stddev" : 1272.220914,
          "N" : 582578,
          "percentile" : {
            "1.000000" : 8640,
            "5.000000" : 8640,
            "10.000000" : 8768,
            "20.000000" : 8768,
            "30.000000" : 8768,
            "40.000000" : 8768,
            "50.000000" : 8768,
            "60.000000" : 8896,
            "70.000000" : 8896,
            "80.000000" : 8896,
            "90.000000" : 9024,
            "95.000000" : 9152,
            "99.000000" : 15680,
            "99.500000" : 16320,
            "99.900000" : 19072,
            "99.950000" : 21888,
            "99.990000" : 55552
          }
        },
        "lat_ns" : {
          "min" : 8035,
          "max" : 110557,
          "mean" : 11410.728043,
          "stddev" : 1289.083345,
          "N" : 582578
        },
        "bw_min" : 41479,
        "bw_max" : 41783,
        "bw_agg" : 100.000000,
        "bw_mean" : 41617.071429,
        "bw_dev" : 75.011245,
        "bw_samples" : 14,
        "iops_min" : 82958,
        "iops_max" : 83566,
        "iops_mean" : 83234.142857,
        "iops_stddev" : 150.022489,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.700000,
      "sys_cpu" : 46.957143,
      "ctx" : 582454,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.381638,
        "20" : 2.541119,
        "50" : 0.052868,
        "100" : 0.013389,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=138675: Wed Oct 26 04:22:19 2022
  read: IOPS=83.2k, BW=40.6MiB/s (42.6MB/s)(284MiB/7001msec)
    slat (nsec): min=2224, max=28664, avg=2408.06, stdev=216.25
    clat (nsec): min=171, max=108203, avg=9002.67, stdev=1272.22
     lat (usec): min=8, max=110, avg=11.41, stdev= 1.29
    clat percentiles (nsec):
     |  1.00th=[ 8640],  5.00th=[ 8640], 10.00th=[ 8768], 20.00th=[ 8768],
     | 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8896],
     | 70.00th=[ 8896], 80.00th=[ 8896], 90.00th=[ 9024], 95.00th=[ 9152],
     | 99.00th=[15680], 99.50th=[16320], 99.90th=[19072], 99.95th=[21888],
     | 99.99th=[55552]
   bw (  KiB/s): min=41479, max=41783, per=100.00%, avg=41617.07, stdev=75.01, samples=14
   iops        : min=82958, max=83566, avg=83234.14, stdev=150.02, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=97.38%, 20=2.54%, 50=0.05%, 100=0.01%
  lat (usec)   : 250=0.01%
  cpu          : usr=11.70%, sys=46.96%, ctx=582454, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=582578,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=40.6MiB/s (42.6MB/s), 40.6MiB/s-40.6MiB/s (42.6MB/s-42.6MB/s), io=284MiB (298MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666750953,
  "timestamp_ms" : 1666750953459,
  "time" : "Wed Oct 26 04:22:33 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 299215872,
        "io_kbytes" : 292203,
        "bw_bytes" : 42739018,
        "bw" : 41737,
        "iops" : 83474.646479,
        "runtime" : 7001,
        "total_ios" : 584406,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2245,
          "max" : 30237,
          "mean" : 2410.540881,
          "stddev" : 223.499942,
          "N" : 584406
        },
        "clat_ns" : {
          "min" : 190,
          "max" : 117380,
          "mean" : 8968.711232,
          "stddev" : 1341.037263,
          "N" : 584406,
          "percentile" : {
            "1.000000" : 8640,
            "5.000000" : 8640,
            "10.000000" : 8640,
            "20.000000" : 8768,
            "30.000000" : 8768,
            "40.000000" : 8768,
            "50.000000" : 8768,
            "60.000000" : 8768,
            "70.000000" : 8768,
            "80.000000" : 8896,
            "90.000000" : 8896,
            "95.000000" : 9024,
            "99.000000" : 15680,
            "99.500000" : 16512,
            "99.900000" : 19072,
            "99.950000" : 22912,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 8446,
          "max" : 119784,
          "mean" : 11379.252112,
          "stddev" : 1357.748455,
          "N" : 584406
        },
        "bw_min" : 41568,
        "bw_max" : 41880,
        "bw_agg" : 100.000000,
        "bw_mean" : 41747.214286,
        "bw_dev" : 89.871856,
        "bw_samples" : 14,
        "iops_min" : 83136,
        "iops_max" : 83760,
        "iops_mean" : 83494.500000,
        "iops_stddev" : 179.757422,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.600000,
      "sys_cpu" : 47.971429,
      "ctx" : 584275,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 97.434147,
        "20" : 2.479270,
        "50" : 0.057152,
        "100" : 0.018651,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=138713: Wed Oct 26 04:22:33 2022
  read: IOPS=83.5k, BW=40.8MiB/s (42.7MB/s)(285MiB/7001msec)
    slat (nsec): min=2245, max=30237, avg=2410.54, stdev=223.50
    clat (nsec): min=190, max=117380, avg=8968.71, stdev=1341.04
     lat (usec): min=8, max=119, avg=11.38, stdev= 1.36
    clat percentiles (nsec):
     |  1.00th=[ 8640],  5.00th=[ 8640], 10.00th=[ 8640], 20.00th=[ 8768],
     | 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8768],
     | 70.00th=[ 8768], 80.00th=[ 8896], 90.00th=[ 8896], 95.00th=[ 9024],
     | 99.00th=[15680], 99.50th=[16512], 99.90th=[19072], 99.95th=[22912],
     | 99.99th=[57088]
   bw (  KiB/s): min=41568, max=41880, per=100.00%, avg=41747.21, stdev=89.87, samples=14
   iops        : min=83136, max=83760, avg=83494.50, stdev=179.76, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 10=97.43%, 20=2.48%, 50=0.06%, 100=0.02%, 250=0.01%
  cpu          : usr=10.60%, sys=47.97%, ctx=584275, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=584406,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=40.8MiB/s (42.7MB/s), 40.8MiB/s-40.8MiB/s (42.7MB/s-42.7MB/s), io=285MiB (299MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666750967,
  "timestamp_ms" : 1666750967372,
  "time" : "Wed Oct 26 04:22:47 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 299606016,
        "io_kbytes" : 292584,
        "bw_bytes" : 42794745,
        "bw" : 41791,
        "iops" : 83583.345236,
        "runtime" : 7001,
        "total_ios" : 585167,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2244,
          "max" : 36078,
          "mean" : 2458.829481,
          "stddev" : 222.580918,
          "N" : 585168
        },
        "clat_ns" : {
          "min" : 170,
          "max" : 108393,
          "mean" : 8930.150774,
          "stddev" : 1341.344004,
          "N" : 585167,
          "percentile" : {
            "1.000000" : 8512,
            "5.000000" : 8640,
            "10.000000" : 8640,
            "20.000000" : 8640,
            "30.000000" : 8640,
            "40.000000" : 8768,
            "50.000000" : 8768,
            "60.000000" : 8768,
            "70.000000" : 8768,
            "80.000000" : 8768,
            "90.000000" : 8896,
            "95.000000" : 9024,
            "99.000000" : 15680,
            "99.500000" : 16320,
            "99.900000" : 19072,
            "99.950000" : 22656,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 8526,
          "max" : 110867,
          "mean" : 11388.980330,
          "stddev" : 1357.766479,
          "N" : 585167
        },
        "bw_min" : 41636,
        "bw_max" : 41936,
        "bw_agg" : 100.000000,
        "bw_mean" : 41802.000000,
        "bw_dev" : 76.181968,
        "bw_samples" : 14,
        "iops_min" : 83272,
        "iops_max" : 83872,
        "iops_mean" : 83604.071429,
        "iops_stddev" : 152.343975,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.771429,
      "sys_cpu" : 45.685714,
      "ctx" : 585039,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 97.435775,
        "20" : 2.483394,
        "50" : 0.051097,
        "100" : 0.018285,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=138751: Wed Oct 26 04:22:47 2022
  read: IOPS=83.6k, BW=40.8MiB/s (42.8MB/s)(286MiB/7001msec)
    slat (nsec): min=2244, max=36078, avg=2458.83, stdev=222.58
    clat (nsec): min=170, max=108393, avg=8930.15, stdev=1341.34
     lat (usec): min=8, max=110, avg=11.39, stdev= 1.36
    clat percentiles (nsec):
     |  1.00th=[ 8512],  5.00th=[ 8640], 10.00th=[ 8640], 20.00th=[ 8640],
     | 30.00th=[ 8640], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8768],
     | 70.00th=[ 8768], 80.00th=[ 8768], 90.00th=[ 8896], 95.00th=[ 9024],
     | 99.00th=[15680], 99.50th=[16320], 99.90th=[19072], 99.95th=[22656],
     | 99.99th=[57088]
   bw (  KiB/s): min=41636, max=41936, per=100.00%, avg=41802.00, stdev=76.18, samples=14
   iops        : min=83272, max=83872, avg=83604.07, stdev=152.34, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 10=97.44%, 20=2.48%, 50=0.05%, 100=0.02%, 250=0.01%
  cpu          : usr=12.77%, sys=45.69%, ctx=585039, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=585167,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=40.8MiB/s (42.8MB/s), 40.8MiB/s-40.8MiB/s (42.8MB/s-42.8MB/s), io=286MiB (300MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt
$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751411,
  "timestamp_ms" : 1666751411748,
  "time" : "Wed Oct 26 04:30:11 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 737202688,
        "io_kbytes" : 719924,
        "bw_bytes" : 105299626,
        "bw" : 102831,
        "iops" : 205663.190973,
        "runtime" : 7001,
        "total_ios" : 1439848,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1152,
          "max" : 27401,
          "mean" : 1324.465338,
          "stddev" : 150.281294,
          "N" : 1439848
        },
        "clat_ns" : {
          "min" : 80,
          "max" : 101000,
          "mean" : 8115.300295,
          "stddev" : 2223.862930,
          "N" : 1439849,
          "percentile" : {
            "1.000000" : 6048,
            "5.000000" : 6112,
            "10.000000" : 6112,
            "20.000000" : 6176,
            "30.000000" : 6240,
            "40.000000" : 6304,
            "50.000000" : 8768,
            "60.000000" : 9536,
            "70.000000" : 9536,
            "80.000000" : 9664,
            "90.000000" : 9664,
            "95.000000" : 9792,
            "99.000000" : 16512,
            "99.500000" : 17280,
            "99.900000" : 18560,
            "99.950000" : 22656,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5911,
          "max" : 102202,
          "mean" : 9439.765673,
          "stddev" : 2290.846561,
          "N" : 1439849
        },
        "bw_min" : 102294,
        "bw_max" : 103753,
        "bw_agg" : 100.000000,
        "bw_mean" : 102856.357143,
        "bw_dev" : 489.366562,
        "bw_samples" : 14,
        "iops_min" : 204588,
        "iops_max" : 207506,
        "iops_mean" : 205712.714286,
        "iops_stddev" : 978.733124,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 18.900000,
      "sys_cpu" : 50.014286,
      "ctx" : 719850,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 96.593182,
        "20" : 3.315558,
        "50" : 0.057784,
        "100" : 0.023961,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=140132: Wed Oct 26 04:30:11 2022
  read: IOPS=206k, BW=100MiB/s (105MB/s)(703MiB/7001msec)
    slat (nsec): min=1152, max=27401, avg=1324.47, stdev=150.28
    clat (nsec): min=80, max=101000, avg=8115.30, stdev=2223.86
     lat (usec): min=5, max=102, avg= 9.44, stdev= 2.29
    clat percentiles (nsec):
     |  1.00th=[ 6048],  5.00th=[ 6112], 10.00th=[ 6112], 20.00th=[ 6176],
     | 30.00th=[ 6240], 40.00th=[ 6304], 50.00th=[ 8768], 60.00th=[ 9536],
     | 70.00th=[ 9536], 80.00th=[ 9664], 90.00th=[ 9664], 95.00th=[ 9792],
     | 99.00th=[16512], 99.50th=[17280], 99.90th=[18560], 99.95th=[22656],
     | 99.99th=[61184]
   bw (  KiB/s): min=102294, max=103753, per=100.00%, avg=102856.36, stdev=489.37, samples=14
   iops        : min=204588, max=207506, avg=205712.71, stdev=978.73, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=96.59%, 20=3.32%, 50=0.06%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=18.90%, sys=50.01%, ctx=719850, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1439848,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=703MiB (737MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751425,
  "timestamp_ms" : 1666751425348,
  "time" : "Wed Oct 26 04:30:25 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 741398528,
        "io_kbytes" : 724022,
        "bw_bytes" : 105898947,
        "bw" : 103416,
        "iops" : 206833.738037,
        "runtime" : 7001,
        "total_ios" : 1448043,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1142,
          "max" : 40395,
          "mean" : 1297.229181,
          "stddev" : 146.300890,
          "N" : 1448043
        },
        "clat_ns" : {
          "min" : 80,
          "max" : 96732,
          "mean" : 8087.446961,
          "stddev" : 2216.390509,
          "N" : 1448044,
          "percentile" : {
            "1.000000" : 5984,
            "5.000000" : 6048,
            "10.000000" : 6112,
            "20.000000" : 6176,
            "30.000000" : 6240,
            "40.000000" : 6304,
            "50.000000" : 8096,
            "60.000000" : 9408,
            "70.000000" : 9536,
            "80.000000" : 9536,
            "90.000000" : 9664,
            "95.000000" : 9792,
            "99.000000" : 16320,
            "99.500000" : 17280,
            "99.900000" : 19072,
            "99.950000" : 26752,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 6071,
          "max" : 97904,
          "mean" : 9384.676166,
          "stddev" : 2292.860984,
          "N" : 1448044
        },
        "bw_min" : 103177,
        "bw_max" : 103737,
        "bw_agg" : 100.000000,
        "bw_mean" : 103442.142857,
        "bw_dev" : 188.649801,
        "bw_samples" : 14,
        "iops_min" : 206354,
        "iops_max" : 207474,
        "iops_mean" : 206884.285714,
        "iops_stddev" : 377.119330,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.357143,
      "sys_cpu" : 51.185714,
      "ctx" : 723965,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 96.716948,
        "20" : 3.182640,
        "50" : 0.069128,
        "100" : 0.023135,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=140171: Wed Oct 26 04:30:25 2022
  read: IOPS=207k, BW=101MiB/s (106MB/s)(707MiB/7001msec)
    slat (nsec): min=1142, max=40395, avg=1297.23, stdev=146.30
    clat (nsec): min=80, max=96732, avg=8087.45, stdev=2216.39
     lat (nsec): min=6071, max=97904, avg=9384.68, stdev=2292.86
    clat percentiles (nsec):
     |  1.00th=[ 5984],  5.00th=[ 6048], 10.00th=[ 6112], 20.00th=[ 6176],
     | 30.00th=[ 6240], 40.00th=[ 6304], 50.00th=[ 8096], 60.00th=[ 9408],
     | 70.00th=[ 9536], 80.00th=[ 9536], 90.00th=[ 9664], 95.00th=[ 9792],
     | 99.00th=[16320], 99.50th=[17280], 99.90th=[19072], 99.95th=[26752],
     | 99.99th=[61184]
   bw (  KiB/s): min=103177, max=103737, per=100.00%, avg=103442.14, stdev=188.65, samples=14
   iops        : min=206354, max=207474, avg=206884.29, stdev=377.12, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=96.72%, 20=3.18%, 50=0.07%
  lat (usec)   : 100=0.02%
  cpu          : usr=17.36%, sys=51.19%, ctx=723965, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1448043,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=101MiB/s (106MB/s), 101MiB/s-101MiB/s (106MB/s-106MB/s), io=707MiB (741MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751438,
  "timestamp_ms" : 1666751438950,
  "time" : "Wed Oct 26 04:30:38 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 734948864,
        "io_kbytes" : 717723,
        "bw_bytes" : 104977698,
        "bw" : 102517,
        "iops" : 205034.280817,
        "runtime" : 7001,
        "total_ios" : 1435445,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1182,
          "max" : 27221,
          "mean" : 1330.533098,
          "stddev" : 132.496465,
          "N" : 1435446
        },
        "clat_ns" : {
          "min" : 100,
          "max" : 117019,
          "mean" : 8150.587585,
          "stddev" : 2236.609171,
          "N" : 1435446,
          "percentile" : {
            "1.000000" : 6048,
            "5.000000" : 6112,
            "10.000000" : 6112,
            "20.000000" : 6240,
            "30.000000" : 6304,
            "40.000000" : 6304,
            "50.000000" : 8640,
            "60.000000" : 9536,
            "70.000000" : 9664,
            "80.000000" : 9664,
            "90.000000" : 9792,
            "95.000000" : 9792,
            "99.000000" : 16512,
            "99.500000" : 17280,
            "99.900000" : 20096,
            "99.950000" : 27008,
            "99.990000" : 62208
          }
        },
        "lat_ns" : {
          "min" : 5871,
          "max" : 118241,
          "mean" : 9481.120683,
          "stddev" : 2308.727929,
          "N" : 1435446
        },
        "bw_min" : 102325,
        "bw_max" : 102710,
        "bw_agg" : 100.000000,
        "bw_mean" : 102550.214286,
        "bw_dev" : 111.494173,
        "bw_samples" : 14,
        "iops_min" : 204650,
        "iops_max" : 205420,
        "iops_mean" : 205100.428571,
        "iops_stddev" : 223.226933,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.157143,
      "sys_cpu" : 48.828571,
      "ctx" : 717624,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 96.261577,
        "20" : 3.628561,
        "50" : 0.077885,
        "100" : 0.021875,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=140209: Wed Oct 26 04:30:38 2022
  read: IOPS=205k, BW=100MiB/s (105MB/s)(701MiB/7001msec)
    slat (nsec): min=1182, max=27221, avg=1330.53, stdev=132.50
    clat (nsec): min=100, max=117019, avg=8150.59, stdev=2236.61
     lat (usec): min=5, max=118, avg= 9.48, stdev= 2.31
    clat percentiles (nsec):
     |  1.00th=[ 6048],  5.00th=[ 6112], 10.00th=[ 6112], 20.00th=[ 6240],
     | 30.00th=[ 6304], 40.00th=[ 6304], 50.00th=[ 8640], 60.00th=[ 9536],
     | 70.00th=[ 9664], 80.00th=[ 9664], 90.00th=[ 9792], 95.00th=[ 9792],
     | 99.00th=[16512], 99.50th=[17280], 99.90th=[20096], 99.95th=[27008],
     | 99.99th=[62208]
   bw (  KiB/s): min=102325, max=102710, per=100.00%, avg=102550.21, stdev=111.49, samples=14
   iops        : min=204650, max=205420, avg=205100.43, stdev=223.23, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=96.26%, 20=3.63%, 50=0.08%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=20.16%, sys=48.83%, ctx=717624, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1435445,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=701MiB (735MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751452,
  "timestamp_ms" : 1666751452581,
  "time" : "Wed Oct 26 04:30:52 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 726595072,
        "io_kbytes" : 709565,
        "bw_bytes" : 103784469,
        "bw" : 101352,
        "iops" : 202703.756606,
        "runtime" : 7001,
        "total_ios" : 1419129,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1212,
          "max" : 64080,
          "mean" : 1388.671381,
          "stddev" : 173.652155,
          "N" : 1419130
        },
        "clat_ns" : {
          "min" : 121,
          "max" : 84979,
          "mean" : 8206.406104,
          "stddev" : 2089.729687,
          "N" : 1419130,
          "percentile" : {
            "1.000000" : 5216,
            "5.000000" : 5536,
            "10.000000" : 6304,
            "20.000000" : 6368,
            "30.000000" : 6496,
            "40.000000" : 7776,
            "50.000000" : 8096,
            "60.000000" : 8256,
            "70.000000" : 9792,
            "80.000000" : 9920,
            "90.000000" : 10048,
            "95.000000" : 10176,
            "99.000000" : 16512,
            "99.500000" : 17024,
            "99.900000" : 18560,
            "99.950000" : 22912,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5921,
          "max" : 86271,
          "mean" : 9595.077485,
          "stddev" : 2157.718644,
          "N" : 1419130
        },
        "bw_min" : 101000,
        "bw_max" : 101608,
        "bw_agg" : 100.000000,
        "bw_mean" : 101386.571429,
        "bw_dev" : 192.066380,
        "bw_samples" : 14,
        "iops_min" : 202000,
        "iops_max" : 203216,
        "iops_mean" : 202773.285714,
        "iops_stddev" : 384.198953,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.300000,
      "sys_cpu" : 49.800000,
      "ctx" : 709604,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 87.470413,
        "20" : 12.438193,
        "50" : 0.058275,
        "100" : 0.019730,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=140247: Wed Oct 26 04:30:52 2022
  read: IOPS=203k, BW=99.0MiB/s (104MB/s)(693MiB/7001msec)
    slat (nsec): min=1212, max=64080, avg=1388.67, stdev=173.65
    clat (nsec): min=121, max=84979, avg=8206.41, stdev=2089.73
     lat (nsec): min=5921, max=86271, avg=9595.08, stdev=2157.72
    clat percentiles (nsec):
     |  1.00th=[ 5216],  5.00th=[ 5536], 10.00th=[ 6304], 20.00th=[ 6368],
     | 30.00th=[ 6496], 40.00th=[ 7776], 50.00th=[ 8096], 60.00th=[ 8256],
     | 70.00th=[ 9792], 80.00th=[ 9920], 90.00th=[10048], 95.00th=[10176],
     | 99.00th=[16512], 99.50th=[17024], 99.90th=[18560], 99.95th=[22912],
     | 99.99th=[61184]
   bw (  KiB/s): min=101000, max=101608, per=100.00%, avg=101386.57, stdev=192.07, samples=14
   iops        : min=202000, max=203216, avg=202773.29, stdev=384.20, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=87.47%, 20=12.44%, 50=0.06%
  lat (usec)   : 100=0.02%
  cpu          : usr=19.30%, sys=49.80%, ctx=709604, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1419129,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=99.0MiB/s (104MB/s), 99.0MiB/s-99.0MiB/s (104MB/s-104MB/s), io=693MiB (727MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751466,
  "timestamp_ms" : 1666751466213,
  "time" : "Wed Oct 26 04:31:06 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 722791424,
        "io_kbytes" : 705851,
        "bw_bytes" : 103241168,
        "bw" : 100821,
        "iops" : 201642.765319,
        "runtime" : 7001,
        "total_ios" : 1411701,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1202,
          "max" : 25348,
          "mean" : 1368.416727,
          "stddev" : 146.449095,
          "N" : 1411701
        },
        "clat_ns" : {
          "min" : 120,
          "max" : 109946,
          "mean" : 8258.915002,
          "stddev" : 2093.679574,
          "N" : 1411702,
          "percentile" : {
            "1.000000" : 5472,
            "5.000000" : 6240,
            "10.000000" : 6240,
            "20.000000" : 6304,
            "30.000000" : 6432,
            "40.000000" : 7776,
            "50.000000" : 8032,
            "60.000000" : 8256,
            "70.000000" : 9792,
            "80.000000" : 9920,
            "90.000000" : 9920,
            "95.000000" : 10048,
            "99.000000" : 16512,
            "99.500000" : 17024,
            "99.900000" : 18560,
            "99.950000" : 23680,
            "99.990000" : 60672
          }
        },
        "lat_ns" : {
          "min" : 6092,
          "max" : 111208,
          "mean" : 9627.331782,
          "stddev" : 2159.028715,
          "N" : 1411702
        },
        "bw_min" : 100667,
        "bw_max" : 101169,
        "bw_agg" : 100.000000,
        "bw_mean" : 100850.000000,
        "bw_dev" : 142.687394,
        "bw_samples" : 14,
        "iops_min" : 201334,
        "iops_max" : 202338,
        "iops_mean" : 201700.000000,
        "iops_stddev" : 285.374787,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 21.028571,
      "sys_cpu" : 47.785714,
      "ctx" : 705798,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 91.422546,
        "20" : 8.482674,
        "50" : 0.060565,
        "100" : 0.022597,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=140286: Wed Oct 26 04:31:06 2022
  read: IOPS=202k, BW=98.5MiB/s (103MB/s)(689MiB/7001msec)
    slat (nsec): min=1202, max=25348, avg=1368.42, stdev=146.45
    clat (nsec): min=120, max=109946, avg=8258.92, stdev=2093.68
     lat (usec): min=6, max=111, avg= 9.63, stdev= 2.16
    clat percentiles (nsec):
     |  1.00th=[ 5472],  5.00th=[ 6240], 10.00th=[ 6240], 20.00th=[ 6304],
     | 30.00th=[ 6432], 40.00th=[ 7776], 50.00th=[ 8032], 60.00th=[ 8256],
     | 70.00th=[ 9792], 80.00th=[ 9920], 90.00th=[ 9920], 95.00th=[10048],
     | 99.00th=[16512], 99.50th=[17024], 99.90th=[18560], 99.95th=[23680],
     | 99.99th=[60672]
   bw (  KiB/s): min=100667, max=101169, per=100.00%, avg=100850.00, stdev=142.69, samples=14
   iops        : min=201334, max=202338, avg=201700.00, stdev=285.37, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=91.42%, 20=8.48%, 50=0.06%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=21.03%, sys=47.79%, ctx=705798, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1411701,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=98.5MiB/s (103MB/s), 98.5MiB/s-98.5MiB/s (103MB/s-103MB/s), io=689MiB (723MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751479,
  "timestamp_ms" : 1666751479857,
  "time" : "Wed Oct 26 04:31:19 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 727922688,
        "io_kbytes" : 710862,
        "bw_bytes" : 103974101,
        "bw" : 101537,
        "iops" : 203074.275104,
        "runtime" : 7001,
        "total_ios" : 1421723,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1212,
          "max" : 24436,
          "mean" : 1384.957250,
          "stddev" : 152.943213,
          "N" : 1421723
        },
        "clat_ns" : {
          "min" : 120,
          "max" : 77255,
          "mean" : 8203.277642,
          "stddev" : 2113.140529,
          "N" : 1421724,
          "percentile" : {
            "1.000000" : 5280,
            "5.000000" : 6176,
            "10.000000" : 6240,
            "20.000000" : 6304,
            "30.000000" : 6432,
            "40.000000" : 7776,
            "50.000000" : 8032,
            "60.000000" : 8256,
            "70.000000" : 9792,
            "80.000000" : 9920,
            "90.000000" : 9920,
            "95.000000" : 10048,
            "99.000000" : 16320,
            "99.500000" : 17280,
            "99.900000" : 18816,
            "99.950000" : 23424,
            "99.990000" : 60672
          }
        },
        "lat_ns" : {
          "min" : 6181,
          "max" : 78517,
          "mean" : 9588.234953,
          "stddev" : 2180.631735,
          "N" : 1421724
        },
        "bw_min" : 101351,
        "bw_max" : 101836,
        "bw_agg" : 100.000000,
        "bw_mean" : 101561.714286,
        "bw_dev" : 140.942666,
        "bw_samples" : 14,
        "iops_min" : 202702,
        "iops_max" : 203672,
        "iops_mean" : 203123.428571,
        "iops_stddev" : 281.885332,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.828571,
      "sys_cpu" : 49.057143,
      "ctx" : 710806,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 92.513310,
        "20" : 7.385335,
        "50" : 0.065695,
        "100" : 0.023774,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=140325: Wed Oct 26 04:31:19 2022
  read: IOPS=203k, BW=99.2MiB/s (104MB/s)(694MiB/7001msec)
    slat (nsec): min=1212, max=24436, avg=1384.96, stdev=152.94
    clat (nsec): min=120, max=77255, avg=8203.28, stdev=2113.14
     lat (nsec): min=6181, max=78517, avg=9588.23, stdev=2180.63
    clat percentiles (nsec):
     |  1.00th=[ 5280],  5.00th=[ 6176], 10.00th=[ 6240], 20.00th=[ 6304],
     | 30.00th=[ 6432], 40.00th=[ 7776], 50.00th=[ 8032], 60.00th=[ 8256],
     | 70.00th=[ 9792], 80.00th=[ 9920], 90.00th=[ 9920], 95.00th=[10048],
     | 99.00th=[16320], 99.50th=[17280], 99.90th=[18816], 99.95th=[23424],
     | 99.99th=[60672]
   bw (  KiB/s): min=101351, max=101836, per=100.00%, avg=101561.71, stdev=140.94, samples=14
   iops        : min=202702, max=203672, avg=203123.43, stdev=281.89, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=92.51%, 20=7.39%, 50=0.07%
  lat (usec)   : 100=0.02%
  cpu          : usr=19.83%, sys=49.06%, ctx=710806, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1421723,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=99.2MiB/s (104MB/s), 99.2MiB/s-99.2MiB/s (104MB/s-104MB/s), io=694MiB (728MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666750981,
  "timestamp_ms" : 1666750981040,
  "time" : "Wed Oct 26 04:23:01 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 290464256,
        "io_kbytes" : 283656,
        "bw_bytes" : 41488966,
        "bw" : 40516,
        "iops" : 81032.995286,
        "runtime" : 7001,
        "total_ios" : 567312,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2304,
          "max" : 40236,
          "mean" : 2576.996690,
          "stddev" : 232.578518,
          "N" : 567313
        },
        "clat_ns" : {
          "min" : 340,
          "max" : 70823,
          "mean" : 9183.791792,
          "stddev" : 1245.876786,
          "N" : 567312,
          "percentile" : {
            "1.000000" : 8768,
            "5.000000" : 8896,
            "10.000000" : 8896,
            "20.000000" : 8896,
            "30.000000" : 8896,
            "40.000000" : 9024,
            "50.000000" : 9024,
            "60.000000" : 9024,
            "70.000000" : 9024,
            "80.000000" : 9024,
            "90.000000" : 9152,
            "95.000000" : 9280,
            "99.000000" : 15552,
            "99.500000" : 16512,
            "99.900000" : 19328,
            "99.950000" : 21632,
            "99.990000" : 55552
          }
        },
        "lat_ns" : {
          "min" : 7985,
          "max" : 73398,
          "mean" : 11760.788467,
          "stddev" : 1263.053401,
          "N" : 567312
        },
        "bw_min" : 40426,
        "bw_max" : 40799,
        "bw_agg" : 100.000000,
        "bw_mean" : 40525.857143,
        "bw_dev" : 100.937038,
        "bw_samples" : 14,
        "iops_min" : 80852,
        "iops_max" : 81598,
        "iops_mean" : 81051.714286,
        "iops_stddev" : 201.874077,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.071429,
      "sys_cpu" : 45.314286,
      "ctx" : 567166,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.013220,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 97.317525,
        "20" : 2.597512,
        "50" : 0.056583,
        "100" : 0.014983,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=138790: Wed Oct 26 04:23:01 2022
  read: IOPS=81.0k, BW=39.6MiB/s (41.5MB/s)(277MiB/7001msec)
    slat (nsec): min=2304, max=40236, avg=2577.00, stdev=232.58
    clat (nsec): min=340, max=70823, avg=9183.79, stdev=1245.88
     lat (nsec): min=7985, max=73398, avg=11760.79, stdev=1263.05
    clat percentiles (nsec):
     |  1.00th=[ 8768],  5.00th=[ 8896], 10.00th=[ 8896], 20.00th=[ 8896],
     | 30.00th=[ 8896], 40.00th=[ 9024], 50.00th=[ 9024], 60.00th=[ 9024],
     | 70.00th=[ 9024], 80.00th=[ 9024], 90.00th=[ 9152], 95.00th=[ 9280],
     | 99.00th=[15552], 99.50th=[16512], 99.90th=[19328], 99.95th=[21632],
     | 99.99th=[55552]
   bw (  KiB/s): min=40426, max=40799, per=100.00%, avg=40525.86, stdev=100.94, samples=14
   iops        : min=80852, max=81598, avg=81051.71, stdev=201.87, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 10=97.32%, 20=2.60%, 50=0.06%, 100=0.01%
  cpu          : usr=14.07%, sys=45.31%, ctx=567166, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=567312,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=39.6MiB/s (41.5MB/s), 39.6MiB/s-39.6MiB/s (41.5MB/s-41.5MB/s), io=277MiB (290MB), run=7001-7001msec

$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751494,
  "timestamp_ms" : 1666751494875,
  "time" : "Wed Oct 26 04:31:34 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 840442368,
        "io_kbytes" : 820744,
        "bw_bytes" : 120046045,
        "bw" : 117232,
        "iops" : 234464.790744,
        "runtime" : 7001,
        "total_ios" : 1641488,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 230,
          "max" : 20989,
          "mean" : 273.106698,
          "stddev" : 42.767660,
          "N" : 1641488
        },
        "clat_ns" : {
          "min" : 270,
          "max" : 112260,
          "mean" : 8033.977383,
          "stddev" : 2024.239647,
          "N" : 1641489,
          "percentile" : {
            "1.000000" : 6368,
            "5.000000" : 6432,
            "10.000000" : 6496,
            "20.000000" : 6944,
            "30.000000" : 7392,
            "40.000000" : 7456,
            "50.000000" : 7648,
            "60.000000" : 7840,
            "70.000000" : 8640,
            "80.000000" : 8768,
            "90.000000" : 8896,
            "95.000000" : 9024,
            "99.000000" : 18816,
            "99.500000" : 20096,
            "99.900000" : 22400,
            "99.950000" : 27776,
            "99.990000" : 62208
          }
        },
        "lat_ns" : {
          "min" : 6081,
          "max" : 112541,
          "mean" : 8307.084073,
          "stddev" : 2025.511582,
          "N" : 1641489
        },
        "bw_min" : 116935,
        "bw_max" : 117650,
        "bw_agg" : 100.000000,
        "bw_mean" : 117259.714286,
        "bw_dev" : 195.708350,
        "bw_samples" : 14,
        "iops_min" : 233870,
        "iops_max" : 235300,
        "iops_mean" : 234519.428571,
        "iops_stddev" : 391.416700,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 25,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.981480,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 96.879478,
        "20" : 2.601055,
        "50" : 0.497293,
        "100" : 0.021505,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140374: Wed Oct 26 04:31:34 2022
  read: IOPS=234k, BW=114MiB/s (120MB/s)(802MiB/7001msec)
    slat (nsec): min=230, max=20989, avg=273.11, stdev=42.77
    clat (nsec): min=270, max=112260, avg=8033.98, stdev=2024.24
     lat (usec): min=6, max=112, avg= 8.31, stdev= 2.03
    clat percentiles (nsec):
     |  1.00th=[ 6368],  5.00th=[ 6432], 10.00th=[ 6496], 20.00th=[ 6944],
     | 30.00th=[ 7392], 40.00th=[ 7456], 50.00th=[ 7648], 60.00th=[ 7840],
     | 70.00th=[ 8640], 80.00th=[ 8768], 90.00th=[ 8896], 95.00th=[ 9024],
     | 99.00th=[18816], 99.50th=[20096], 99.90th=[22400], 99.95th=[27776],
     | 99.99th=[62208]
   bw (  KiB/s): min=116935, max=117650, per=100.00%, avg=117259.71, stdev=195.71, samples=14
   iops        : min=233870, max=235300, avg=234519.43, stdev=391.42, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=96.88%, 20=2.60%, 50=0.50%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=25, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1641488,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=114MiB/s (120MB/s), 114MiB/s-114MiB/s (120MB/s-120MB/s), io=802MiB (840MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751509,
  "timestamp_ms" : 1666751509970,
  "time" : "Wed Oct 26 04:31:49 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 906781696,
        "io_kbytes" : 885529,
        "bw_bytes" : 129521739,
        "bw" : 126486,
        "iops" : 252972.003999,
        "runtime" : 7001,
        "total_ios" : 1771057,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 210,
          "max" : 10360,
          "mean" : 266.127212,
          "stddev" : 24.148686,
          "N" : 1771057
        },
        "clat_ns" : {
          "min" : 281,
          "max" : 77735,
          "mean" : 7425.552159,
          "stddev" : 1737.669617,
          "N" : 1771058,
          "percentile" : {
            "1.000000" : 6816,
            "5.000000" : 7008,
            "10.000000" : 7008,
            "20.000000" : 7072,
            "30.000000" : 7072,
            "40.000000" : 7136,
            "50.000000" : 7200,
            "60.000000" : 7200,
            "70.000000" : 7264,
            "80.000000" : 7328,
            "90.000000" : 7392,
            "95.000000" : 7520,
            "99.000000" : 16320,
            "99.500000" : 20864,
            "99.900000" : 22144,
            "99.950000" : 22912,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 5530,
          "max" : 77996,
          "mean" : 7691.679367,
          "stddev" : 1738.017364,
          "N" : 1771058
        },
        "bw_min" : 126151,
        "bw_max" : 126861,
        "bw_agg" : 100.000000,
        "bw_mean" : 126549.785714,
        "bw_dev" : 246.186039,
        "bw_samples" : 14,
        "iops_min" : 252302,
        "iops_max" : 253722,
        "iops_mean" : 253099.642857,
        "iops_stddev" : 492.334097,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.990401,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.000831,
        "20" : 2.473890,
        "50" : 0.499589,
        "100" : 0.025070,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140424: Wed Oct 26 04:31:49 2022
  read: IOPS=253k, BW=124MiB/s (130MB/s)(865MiB/7001msec)
    slat (nsec): min=210, max=10360, avg=266.13, stdev=24.15
    clat (nsec): min=281, max=77735, avg=7425.55, stdev=1737.67
     lat (nsec): min=5530, max=77996, avg=7691.68, stdev=1738.02
    clat percentiles (nsec):
     |  1.00th=[ 6816],  5.00th=[ 7008], 10.00th=[ 7008], 20.00th=[ 7072],
     | 30.00th=[ 7072], 40.00th=[ 7136], 50.00th=[ 7200], 60.00th=[ 7200],
     | 70.00th=[ 7264], 80.00th=[ 7328], 90.00th=[ 7392], 95.00th=[ 7520],
     | 99.00th=[16320], 99.50th=[20864], 99.90th=[22144], 99.95th=[22912],
     | 99.99th=[66048]
   bw (  KiB/s): min=126151, max=126861, per=100.00%, avg=126549.79, stdev=246.19, samples=14
   iops        : min=252302, max=253722, avg=253099.64, stdev=492.33, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=97.00%, 20=2.47%, 50=0.50%, 100=0.03%
  cpu          : usr=99.99%, sys=0.00%, ctx=23, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1771057,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=865MiB (907MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751525,
  "timestamp_ms" : 1666751525027,
  "time" : "Wed Oct 26 04:32:05 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 834501120,
        "io_kbytes" : 814942,
        "bw_bytes" : 119197417,
        "bw" : 116403,
        "iops" : 232807.313241,
        "runtime" : 7001,
        "total_ios" : 1629884,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 230,
          "max" : 27211,
          "mean" : 278.884964,
          "stddev" : 29.538508,
          "N" : 1629884
        },
        "clat_ns" : {
          "min" : 872,
          "max" : 121678,
          "mean" : 8094.376448,
          "stddev" : 2042.240059,
          "N" : 1629885,
          "percentile" : {
            "1.000000" : 6368,
            "5.000000" : 6432,
            "10.000000" : 6560,
            "20.000000" : 7264,
            "30.000000" : 7392,
            "40.000000" : 7520,
            "50.000000" : 7648,
            "60.000000" : 7904,
            "70.000000" : 8768,
            "80.000000" : 8896,
            "90.000000" : 8896,
            "95.000000" : 9024,
            "99.000000" : 18816,
            "99.500000" : 20096,
            "99.900000" : 22400,
            "99.950000" : 28544,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 5881,
          "max" : 121959,
          "mean" : 8373.261407,
          "stddev" : 2043.377005,
          "N" : 1629885
        },
        "bw_min" : 116102,
        "bw_max" : 116981,
        "bw_agg" : 100.000000,
        "bw_mean" : 116429.500000,
        "bw_dev" : 245.284372,
        "bw_samples" : 14,
        "iops_min" : 232204,
        "iops_max" : 233962,
        "iops_mean" : 232859.357143,
        "iops_stddev" : 490.490744,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.990551,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.857322,
        "20" : 2.646017,
        "50" : 0.471567,
        "100" : 0.024726,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140474: Wed Oct 26 04:32:05 2022
  read: IOPS=233k, BW=114MiB/s (119MB/s)(796MiB/7001msec)
    slat (nsec): min=230, max=27211, avg=278.88, stdev=29.54
    clat (nsec): min=872, max=121678, avg=8094.38, stdev=2042.24
     lat (usec): min=5, max=121, avg= 8.37, stdev= 2.04
    clat percentiles (nsec):
     |  1.00th=[ 6368],  5.00th=[ 6432], 10.00th=[ 6560], 20.00th=[ 7264],
     | 30.00th=[ 7392], 40.00th=[ 7520], 50.00th=[ 7648], 60.00th=[ 7904],
     | 70.00th=[ 8768], 80.00th=[ 8896], 90.00th=[ 8896], 95.00th=[ 9024],
     | 99.00th=[18816], 99.50th=[20096], 99.90th=[22400], 99.95th=[28544],
     | 99.99th=[62720]
   bw (  KiB/s): min=116102, max=116981, per=100.00%, avg=116429.50, stdev=245.28, samples=14
   iops        : min=232204, max=233962, avg=232859.36, stdev=490.49, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 10=96.86%, 20=2.65%, 50=0.47%, 100=0.02%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=23, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1629884,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s), io=796MiB (835MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751538,
  "timestamp_ms" : 1666751538710,
  "time" : "Wed Oct 26 04:32:18 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 712784384,
        "io_kbytes" : 696078,
        "bw_bytes" : 101811796,
        "bw" : 99425,
        "iops" : 198851.021283,
        "runtime" : 7001,
        "total_ios" : 1392156,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1282,
          "max" : 23945,
          "mean" : 1414.855360,
          "stddev" : 133.152742,
          "N" : 1392156
        },
        "clat_ns" : {
          "min" : 80,
          "max" : 100759,
          "mean" : 8354.827444,
          "stddev" : 2236.779986,
          "N" : 1392157,
          "percentile" : {
            "1.000000" : 5536,
            "5.000000" : 6240,
            "10.000000" : 6304,
            "20.000000" : 6368,
            "30.000000" : 6432,
            "40.000000" : 6496,
            "50.000000" : 8256,
            "60.000000" : 9792,
            "70.000000" : 9920,
            "80.000000" : 10048,
            "90.000000" : 10048,
            "95.000000" : 10176,
            "99.000000" : 16512,
            "99.500000" : 17024,
            "99.900000" : 19072,
            "99.950000" : 23168,
            "99.990000" : 60160
          }
        },
        "lat_ns" : {
          "min" : 6342,
          "max" : 102081,
          "mean" : 9769.682853,
          "stddev" : 2306.249536,
          "N" : 1392157
        },
        "bw_min" : 99158,
        "bw_max" : 99923,
        "bw_agg" : 100.000000,
        "bw_mean" : 99467.928571,
        "bw_dev" : 285.062062,
        "bw_samples" : 14,
        "iops_min" : 198316,
        "iops_max" : 199846,
        "iops_mean" : 198936.071429,
        "iops_stddev" : 570.072932,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 18.328571,
      "sys_cpu" : 51.842857,
      "ctx" : 696330,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 81.288448,
        "20" : 18.611635,
        "50" : 0.069820,
        "100" : 0.020687,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2002080,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 10829,
      "write_ticks" : 0,
      "in_queue" : 10829,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=140513: Wed Oct 26 04:32:18 2022
  read: IOPS=199k, BW=97.1MiB/s (102MB/s)(680MiB/7001msec)
    slat (nsec): min=1282, max=23945, avg=1414.86, stdev=133.15
    clat (nsec): min=80, max=100759, avg=8354.83, stdev=2236.78
     lat (usec): min=6, max=102, avg= 9.77, stdev= 2.31
    clat percentiles (nsec):
     |  1.00th=[ 5536],  5.00th=[ 6240], 10.00th=[ 6304], 20.00th=[ 6368],
     | 30.00th=[ 6432], 40.00th=[ 6496], 50.00th=[ 8256], 60.00th=[ 9792],
     | 70.00th=[ 9920], 80.00th=[10048], 90.00th=[10048], 95.00th=[10176],
     | 99.00th=[16512], 99.50th=[17024], 99.90th=[19072], 99.95th=[23168],
     | 99.99th=[60160]
   bw (  KiB/s): min=99158, max=99923, per=100.00%, avg=99467.93, stdev=285.06, samples=14
   iops        : min=198316, max=199846, avg=198936.07, stdev=570.07, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=81.29%, 20=18.61%, 50=0.07%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=18.33%, sys=51.84%, ctx=696330, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1392156,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=97.1MiB/s (102MB/s), 97.1MiB/s-97.1MiB/s (102MB/s-102MB/s), io=680MiB (713MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2002080/0, merge=0/0, ticks=10829/0, in_queue=10829, util=99.09%

$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt': No such file or directory

$ rm /tmp/fio-output_512_2_io_uring-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751552,
  "timestamp_ms" : 1666751552714,
  "time" : "Wed Oct 26 04:32:32 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 713009152,
        "io_kbytes" : 696298,
        "bw_bytes" : 101843901,
        "bw" : 99456,
        "iops" : 198913.726610,
        "runtime" : 7001,
        "total_ios" : 1392595,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1282,
          "max" : 49703,
          "mean" : 1410.760628,
          "stddev" : 136.155381,
          "N" : 1392595
        },
        "clat_ns" : {
          "min" : 80,
          "max" : 102632,
          "mean" : 8358.082344,
          "stddev" : 2240.215117,
          "N" : 1392596,
          "percentile" : {
            "1.000000" : 5536,
            "5.000000" : 6304,
            "10.000000" : 6304,
            "20.000000" : 6368,
            "30.000000" : 6432,
            "40.000000" : 6496,
            "50.000000" : 8256,
            "60.000000" : 9792,
            "70.000000" : 9920,
            "80.000000" : 9920,
            "90.000000" : 10048,
            "95.000000" : 10176,
            "99.000000" : 16512,
            "99.500000" : 17024,
            "99.900000" : 19072,
            "99.950000" : 23168,
            "99.990000" : 65280
          }
        },
        "lat_ns" : {
          "min" : 5671,
          "max" : 103965,
          "mean" : 9768.843017,
          "stddev" : 2302.931642,
          "N" : 1392596
        },
        "bw_min" : 99307,
        "bw_max" : 99675,
        "bw_agg" : 100.000000,
        "bw_mean" : 99497.357143,
        "bw_dev" : 102.195595,
        "bw_samples" : 14,
        "iops_min" : 198614,
        "iops_max" : 199350,
        "iops_mean" : 198994.785714,
        "iops_stddev" : 204.325599,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.071429,
      "sys_cpu" : 51.157143,
      "ctx" : 696426,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 83.180681,
        "20" : 16.725394,
        "50" : 0.061612,
        "100" : 0.023625,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2000153,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 10851,
      "write_ticks" : 0,
      "in_queue" : 10851,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=140551: Wed Oct 26 04:32:32 2022
  read: IOPS=199k, BW=97.1MiB/s (102MB/s)(680MiB/7001msec)
    slat (nsec): min=1282, max=49703, avg=1410.76, stdev=136.16
    clat (nsec): min=80, max=102632, avg=8358.08, stdev=2240.22
     lat (usec): min=5, max=103, avg= 9.77, stdev= 2.30
    clat percentiles (nsec):
     |  1.00th=[ 5536],  5.00th=[ 6304], 10.00th=[ 6304], 20.00th=[ 6368],
     | 30.00th=[ 6432], 40.00th=[ 6496], 50.00th=[ 8256], 60.00th=[ 9792],
     | 70.00th=[ 9920], 80.00th=[ 9920], 90.00th=[10048], 95.00th=[10176],
     | 99.00th=[16512], 99.50th=[17024], 99.90th=[19072], 99.95th=[23168],
     | 99.99th=[65280]
   bw (  KiB/s): min=99307, max=99675, per=100.00%, avg=99497.36, stdev=102.20, samples=14
   iops        : min=198614, max=199350, avg=198994.79, stdev=204.33, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=83.18%, 20=16.73%, 50=0.06%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=19.07%, sys=51.16%, ctx=696426, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1392595,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=97.1MiB/s (102MB/s), 97.1MiB/s-97.1MiB/s (102MB/s-102MB/s), io=680MiB (713MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2000153/0, merge=0/0, ticks=10851/0, in_queue=10851, util=99.09%

$ rm /tmp/fio-output_512_2_io_uring-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751567,
  "timestamp_ms" : 1666751567468,
  "time" : "Wed Oct 26 04:32:47 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 704743424,
        "io_kbytes" : 688226,
        "bw_bytes" : 100663251,
        "bw" : 98303,
        "iops" : 196607.770319,
        "runtime" : 7001,
        "total_ios" : 1376451,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1302,
          "max" : 115346,
          "mean" : 1448.851801,
          "stddev" : 170.761925,
          "N" : 1376451
        },
        "clat_ns" : {
          "min" : 90,
          "max" : 124113,
          "mean" : 8432.095208,
          "stddev" : 2364.959566,
          "N" : 1376452,
          "percentile" : {
            "1.000000" : 5344,
            "5.000000" : 6304,
            "10.000000" : 6304,
            "20.000000" : 6368,
            "30.000000" : 6432,
            "40.000000" : 6496,
            "50.000000" : 8896,
            "60.000000" : 10048,
            "70.000000" : 10048,
            "80.000000" : 10176,
            "90.000000" : 10176,
            "95.000000" : 10304,
            "99.000000" : 16512,
            "99.500000" : 17280,
            "99.900000" : 23168,
            "99.950000" : 23424,
            "99.990000" : 65280
          }
        },
        "lat_ns" : {
          "min" : 6342,
          "max" : 125696,
          "mean" : 9880.947041,
          "stddev" : 2442.322795,
          "N" : 1376452
        },
        "bw_min" : 98131,
        "bw_max" : 98526,
        "bw_agg" : 100.000000,
        "bw_mean" : 98347.384615,
        "bw_dev" : 105.430497,
        "bw_samples" : 13,
        "iops_min" : 196262,
        "iops_max" : 197052,
        "iops_mean" : 196694.846154,
        "iops_stddev" : 210.890116,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 18.671429,
      "sys_cpu" : 52.142857,
      "ctx" : 689064,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 59.597109,
        "20" : 40.243205,
        "50" : 0.122925,
        "100" : 0.025064,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1979948,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 10785,
      "write_ticks" : 0,
      "in_queue" : 10785,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=140589: Wed Oct 26 04:32:47 2022
  read: IOPS=197k, BW=96.0MiB/s (101MB/s)(672MiB/7001msec)
    slat (nsec): min=1302, max=115346, avg=1448.85, stdev=170.76
    clat (nsec): min=90, max=124113, avg=8432.10, stdev=2364.96
     lat (usec): min=6, max=125, avg= 9.88, stdev= 2.44
    clat percentiles (nsec):
     |  1.00th=[ 5344],  5.00th=[ 6304], 10.00th=[ 6304], 20.00th=[ 6368],
     | 30.00th=[ 6432], 40.00th=[ 6496], 50.00th=[ 8896], 60.00th=[10048],
     | 70.00th=[10048], 80.00th=[10176], 90.00th=[10176], 95.00th=[10304],
     | 99.00th=[16512], 99.50th=[17280], 99.90th=[23168], 99.95th=[23424],
     | 99.99th=[65280]
   bw (  KiB/s): min=98131, max=98526, per=100.00%, avg=98347.38, stdev=105.43, samples=13
   iops        : min=196262, max=197052, avg=196694.85, stdev=210.89, samples=13
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=59.60%, 20=40.24%, 50=0.12%
  lat (usec)   : 100=0.03%, 250=0.01%
  cpu          : usr=18.67%, sys=52.14%, ctx=689064, majf=0, minf=1
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1376451,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=96.0MiB/s (101MB/s), 96.0MiB/s-96.0MiB/s (101MB/s-101MB/s), io=672MiB (705MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1979948/0, merge=0/0, ticks=10785/0, in_queue=10785, util=99.09%

$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751582,
  "timestamp_ms" : 1666751582489,
  "time" : "Wed Oct 26 04:33:02 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 814672896,
        "io_kbytes" : 795579,
        "bw_bytes" : 116365218,
        "bw" : 113637,
        "iops" : 227275.674904,
        "runtime" : 7001,
        "total_ios" : 1591157,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 17412,
          "mean" : 151.050024,
          "stddev" : 62.073951,
          "N" : 1591157
        },
        "clat_ns" : {
          "min" : 5881,
          "max" : 120396,
          "mean" : 8186.654414,
          "stddev" : 1593.597298,
          "N" : 1591157,
          "percentile" : {
            "1.000000" : 7328,
            "5.000000" : 7648,
            "10.000000" : 7712,
            "20.000000" : 7776,
            "30.000000" : 7840,
            "40.000000" : 7904,
            "50.000000" : 7968,
            "60.000000" : 8032,
            "70.000000" : 8096,
            "80.000000" : 8160,
            "90.000000" : 8256,
            "95.000000" : 8384,
            "99.000000" : 16512,
            "99.500000" : 19840,
            "99.900000" : 20608,
            "99.950000" : 21120,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 6071,
          "max" : 120566,
          "mean" : 8337.704438,
          "stddev" : 1596.070054,
          "N" : 1591157
        },
        "bw_min" : 113483,
        "bw_max" : 113870,
        "bw_agg" : 100.000000,
        "bw_mean" : 113676.000000,
        "bw_dev" : 107.724006,
        "bw_samples" : 14,
        "iops_min" : 226966,
        "iops_max" : 227740,
        "iops_mean" : 227352.071429,
        "iops_stddev" : 215.540274,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 33.528571,
      "sys_cpu" : 66.442857,
      "ctx" : 20,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.203818,
        "2" : 50.796182,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.850279,
        "20" : 2.729398,
        "50" : 0.397949,
        "100" : 0.022122,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=140637: Wed Oct 26 04:33:02 2022
  read: IOPS=227k, BW=111MiB/s (116MB/s)(777MiB/7001msec)
    slat (nsec): min=110, max=17412, avg=151.05, stdev=62.07
    clat (usec): min=5, max=120, avg= 8.19, stdev= 1.59
     lat (usec): min=6, max=120, avg= 8.34, stdev= 1.60
    clat percentiles (nsec):
     |  1.00th=[ 7328],  5.00th=[ 7648], 10.00th=[ 7712], 20.00th=[ 7776],
     | 30.00th=[ 7840], 40.00th=[ 7904], 50.00th=[ 7968], 60.00th=[ 8032],
     | 70.00th=[ 8096], 80.00th=[ 8160], 90.00th=[ 8256], 95.00th=[ 8384],
     | 99.00th=[16512], 99.50th=[19840], 99.90th=[20608], 99.95th=[21120],
     | 99.99th=[62720]
   bw (  KiB/s): min=113483, max=113870, per=100.00%, avg=113676.00, stdev=107.72, samples=14
   iops        : min=226966, max=227740, avg=227352.07, stdev=215.54, samples=14
  lat (usec)   : 10=96.85%, 20=2.73%, 50=0.40%, 100=0.02%, 250=0.01%
  cpu          : usr=33.53%, sys=66.44%, ctx=20, majf=0, minf=0
  IO depths    : 1=49.2%, 2=50.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1591157,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=111MiB/s (116MB/s), 111MiB/s-111MiB/s (116MB/s-116MB/s), io=777MiB (815MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751597,
  "timestamp_ms" : 1666751597582,
  "time" : "Wed Oct 26 04:33:17 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 821842432,
        "io_kbytes" : 802580,
        "bw_bytes" : 117389291,
        "bw" : 114637,
        "iops" : 229275.817740,
        "runtime" : 7001,
        "total_ios" : 1605160,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 17734,
          "mean" : 149.779642,
          "stddev" : 53.132110,
          "N" : 1605160
        },
        "clat_ns" : {
          "min" : 5751,
          "max" : 97513,
          "mean" : 8131.512809,
          "stddev" : 1581.103388,
          "N" : 1605160,
          "percentile" : {
            "1.000000" : 7264,
            "5.000000" : 7584,
            "10.000000" : 7648,
            "20.000000" : 7712,
            "30.000000" : 7776,
            "40.000000" : 7840,
            "50.000000" : 7904,
            "60.000000" : 7968,
            "70.000000" : 8032,
            "80.000000" : 8096,
            "90.000000" : 8256,
            "95.000000" : 8384,
            "99.000000" : 16512,
            "99.500000" : 19840,
            "99.900000" : 20864,
            "99.950000" : 21120,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 5952,
          "max" : 97693,
          "mean" : 8281.292451,
          "stddev" : 1583.644423,
          "N" : 1605160
        },
        "bw_min" : 114397,
        "bw_max" : 114917,
        "bw_agg" : 100.000000,
        "bw_mean" : 114668.571429,
        "bw_dev" : 150.823745,
        "bw_samples" : 14,
        "iops_min" : 228794,
        "iops_max" : 229834,
        "iops_mean" : 229337.214286,
        "iops_stddev" : 301.690666,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 35.814286,
      "sys_cpu" : 64.157143,
      "ctx" : 21,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.234718,
        "2" : 50.765282,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.891587,
        "20" : 2.622293,
        "50" : 0.465873,
        "100" : 0.020247,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=140686: Wed Oct 26 04:33:17 2022
  read: IOPS=229k, BW=112MiB/s (117MB/s)(784MiB/7001msec)
    slat (nsec): min=110, max=17734, avg=149.78, stdev=53.13
    clat (nsec): min=5751, max=97513, avg=8131.51, stdev=1581.10
     lat (nsec): min=5952, max=97693, avg=8281.29, stdev=1583.64
    clat percentiles (nsec):
     |  1.00th=[ 7264],  5.00th=[ 7584], 10.00th=[ 7648], 20.00th=[ 7712],
     | 30.00th=[ 7776], 40.00th=[ 7840], 50.00th=[ 7904], 60.00th=[ 7968],
     | 70.00th=[ 8032], 80.00th=[ 8096], 90.00th=[ 8256], 95.00th=[ 8384],
     | 99.00th=[16512], 99.50th=[19840], 99.90th=[20864], 99.95th=[21120],
     | 99.99th=[62720]
   bw (  KiB/s): min=114397, max=114917, per=100.00%, avg=114668.57, stdev=150.82, samples=14
   iops        : min=228794, max=229834, avg=229337.21, stdev=301.69, samples=14
  lat (usec)   : 10=96.89%, 20=2.62%, 50=0.47%, 100=0.02%
  cpu          : usr=35.81%, sys=64.16%, ctx=21, majf=0, minf=0
  IO depths    : 1=49.2%, 2=50.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1605160,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=112MiB/s (117MB/s), 112MiB/s-112MiB/s (117MB/s-117MB/s), io=784MiB (822MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-bdev_uring_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751612,
  "timestamp_ms" : 1666751612671,
  "time" : "Wed Oct 26 04:33:32 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 816369664,
        "io_kbytes" : 797236,
        "bw_bytes" : 116607579,
        "bw" : 113874,
        "iops" : 227749.035852,
        "runtime" : 7001,
        "total_ios" : 1594471,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 23354,
          "mean" : 150.289322,
          "stddev" : 61.636434,
          "N" : 1594471
        },
        "clat_ns" : {
          "min" : 5841,
          "max" : 119644,
          "mean" : 8172.569133,
          "stddev" : 1601.463886,
          "N" : 1594471,
          "percentile" : {
            "1.000000" : 7264,
            "5.000000" : 7648,
            "10.000000" : 7712,
            "20.000000" : 7776,
            "30.000000" : 7840,
            "40.000000" : 7904,
            "50.000000" : 7968,
            "60.000000" : 8032,
            "70.000000" : 8032,
            "80.000000" : 8096,
            "90.000000" : 8256,
            "95.000000" : 8384,
            "99.000000" : 16512,
            "99.500000" : 19840,
            "99.900000" : 20864,
            "99.950000" : 21120,
            "99.990000" : 61696
          }
        },
        "lat_ns" : {
          "min" : 6092,
          "max" : 119814,
          "mean" : 8322.858455,
          "stddev" : 1603.977324,
          "N" : 1594471
        },
        "bw_min" : 113642,
        "bw_max" : 114230,
        "bw_agg" : 100.000000,
        "bw_mean" : 113906.642857,
        "bw_dev" : 147.479650,
        "bw_samples" : 14,
        "iops_min" : 227284,
        "iops_max" : 228460,
        "iops_mean" : 227813.357143,
        "iops_stddev" : 295.000810,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.100000,
      "sys_cpu" : 67.871429,
      "ctx" : 27,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.172171,
        "2" : 50.827829,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.864540,
        "20" : 2.690359,
        "50" : 0.421707,
        "100" : 0.023142,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=140736: Wed Oct 26 04:33:32 2022
  read: IOPS=228k, BW=111MiB/s (117MB/s)(779MiB/7001msec)
    slat (nsec): min=110, max=23354, avg=150.29, stdev=61.64
    clat (usec): min=5, max=119, avg= 8.17, stdev= 1.60
     lat (usec): min=6, max=119, avg= 8.32, stdev= 1.60
    clat percentiles (nsec):
     |  1.00th=[ 7264],  5.00th=[ 7648], 10.00th=[ 7712], 20.00th=[ 7776],
     | 30.00th=[ 7840], 40.00th=[ 7904], 50.00th=[ 7968], 60.00th=[ 8032],
     | 70.00th=[ 8032], 80.00th=[ 8096], 90.00th=[ 8256], 95.00th=[ 8384],
     | 99.00th=[16512], 99.50th=[19840], 99.90th=[20864], 99.95th=[21120],
     | 99.99th=[61696]
   bw (  KiB/s): min=113642, max=114230, per=100.00%, avg=113906.64, stdev=147.48, samples=14
   iops        : min=227284, max=228460, avg=227813.36, stdev=295.00, samples=14
  lat (usec)   : 10=96.86%, 20=2.69%, 50=0.42%, 100=0.02%, 250=0.01%
  cpu          : usr=32.10%, sys=67.87%, ctx=27, majf=0, minf=0
  IO depths    : 1=49.2%, 2=50.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1594471,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=111MiB/s (117MB/s), 111MiB/s-111MiB/s (117MB/s-117MB/s), io=779MiB (816MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751627,
  "timestamp_ms" : 1666751627772,
  "time" : "Wed Oct 26 04:33:47 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 954444288,
        "io_kbytes" : 932074,
        "bw_bytes" : 136329708,
        "bw" : 133134,
        "iops" : 266268.675903,
        "runtime" : 7001,
        "total_ios" : 1864147,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1552,
          "max" : 23744,
          "mean" : 2227.662047,
          "stddev" : 687.257728,
          "N" : 1864148
        },
        "clat_ns" : {
          "min" : 241,
          "max" : 115497,
          "mean" : 5044.965217,
          "stddev" : 1900.368884,
          "N" : 1864148,
          "percentile" : {
            "1.000000" : 4048,
            "5.000000" : 4080,
            "10.000000" : 4128,
            "20.000000" : 4128,
            "30.000000" : 4192,
            "40.000000" : 4256,
            "50.000000" : 5216,
            "60.000000" : 5280,
            "70.000000" : 5344,
            "80.000000" : 5408,
            "90.000000" : 5472,
            "95.000000" : 5664,
            "99.000000" : 14272,
            "99.500000" : 18304,
            "99.900000" : 22400,
            "99.950000" : 22912,
            "99.990000" : 60672
          }
        },
        "lat_ns" : {
          "min" : 5591,
          "max" : 117120,
          "mean" : 7272.627264,
          "stddev" : 1826.947438,
          "N" : 1864148
        },
        "bw_min" : 133014,
        "bw_max" : 133288,
        "bw_agg" : 100.000000,
        "bw_mean" : 133165.500000,
        "bw_dev" : 95.984574,
        "bw_samples" : 14,
        "iops_min" : 266028,
        "iops_max" : 266576,
        "iops_mean" : 266331.000000,
        "iops_stddev" : 191.778318,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 33.700000,
      "sys_cpu" : 66.271429,
      "ctx" : 25,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.022799,
        "10" : 97.619233,
        "20" : 1.967710,
        "50" : 0.368265,
        "100" : 0.019741,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140785: Wed Oct 26 04:33:47 2022
  read: IOPS=266k, BW=130MiB/s (136MB/s)(910MiB/7001msec)
    slat (nsec): min=1552, max=23744, avg=2227.66, stdev=687.26
    clat (nsec): min=241, max=115497, avg=5044.97, stdev=1900.37
     lat (usec): min=5, max=117, avg= 7.27, stdev= 1.83
    clat percentiles (nsec):
     |  1.00th=[ 4048],  5.00th=[ 4080], 10.00th=[ 4128], 20.00th=[ 4128],
     | 30.00th=[ 4192], 40.00th=[ 4256], 50.00th=[ 5216], 60.00th=[ 5280],
     | 70.00th=[ 5344], 80.00th=[ 5408], 90.00th=[ 5472], 95.00th=[ 5664],
     | 99.00th=[14272], 99.50th=[18304], 99.90th=[22400], 99.95th=[22912],
     | 99.99th=[60672]
   bw (  KiB/s): min=133014, max=133288, per=100.00%, avg=133165.50, stdev=95.98, samples=14
   iops        : min=266028, max=266576, avg=266331.00, stdev=191.78, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.02%, 10=97.62%, 20=1.97%, 50=0.37%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=33.70%, sys=66.27%, ctx=25, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1864147,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=910MiB (954MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666750994,
  "timestamp_ms" : 1666750994677,
  "time" : "Wed Oct 26 04:23:14 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 285780992,
        "io_kbytes" : 279083,
        "bw_bytes" : 40820024,
        "bw" : 39863,
        "iops" : 79726.467647,
        "runtime" : 7001,
        "total_ios" : 558165,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2425,
          "max" : 28102,
          "mean" : 2654.688612,
          "stddev" : 235.256413,
          "N" : 558166
        },
        "clat_ns" : {
          "min" : 271,
          "max" : 80301,
          "mean" : 9230.039091,
          "stddev" : 1260.026978,
          "N" : 558165,
          "percentile" : {
            "1.000000" : 8896,
            "5.000000" : 8896,
            "10.000000" : 8896,
            "20.000000" : 9024,
            "30.000000" : 9024,
            "40.000000" : 9024,
            "50.000000" : 9024,
            "60.000000" : 9024,
            "70.000000" : 9152,
            "80.000000" : 9152,
            "90.000000" : 9152,
            "95.000000" : 9408,
            "99.000000" : 15296,
            "99.500000" : 16064,
            "99.900000" : 19328,
            "99.950000" : 27520,
            "99.990000" : 55552
          }
        },
        "lat_ns" : {
          "min" : 9147,
          "max" : 82936,
          "mean" : 11884.727846,
          "stddev" : 1280.549295,
          "N" : 558165
        },
        "bw_min" : 39774,
        "bw_max" : 40040,
        "bw_agg" : 100.000000,
        "bw_mean" : 39873.642857,
        "bw_dev" : 66.964522,
        "bw_samples" : 14,
        "iops_min" : 79548,
        "iops_max" : 80080,
        "iops_mean" : 79747.285714,
        "iops_stddev" : 133.929045,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.842857,
      "sys_cpu" : 45.942857,
      "ctx" : 558046,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 97.336988,
        "20" : 2.566087,
        "50" : 0.071663,
        "100" : 0.015587,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=138828: Wed Oct 26 04:23:14 2022
  read: IOPS=79.7k, BW=38.9MiB/s (40.8MB/s)(273MiB/7001msec)
    slat (nsec): min=2425, max=28102, avg=2654.69, stdev=235.26
    clat (nsec): min=271, max=80301, avg=9230.04, stdev=1260.03
     lat (nsec): min=9147, max=82936, avg=11884.73, stdev=1280.55
    clat percentiles (nsec):
     |  1.00th=[ 8896],  5.00th=[ 8896], 10.00th=[ 8896], 20.00th=[ 9024],
     | 30.00th=[ 9024], 40.00th=[ 9024], 50.00th=[ 9024], 60.00th=[ 9024],
     | 70.00th=[ 9152], 80.00th=[ 9152], 90.00th=[ 9152], 95.00th=[ 9408],
     | 99.00th=[15296], 99.50th=[16064], 99.90th=[19328], 99.95th=[27520],
     | 99.99th=[55552]
   bw (  KiB/s): min=39774, max=40040, per=100.00%, avg=39873.64, stdev=66.96, samples=14
   iops        : min=79548, max=80080, avg=79747.29, stdev=133.93, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=97.34%, 20=2.57%, 50=0.07%, 100=0.02%
  cpu          : usr=13.84%, sys=45.94%, ctx=558046, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=558165,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=38.9MiB/s (40.8MB/s), 38.9MiB/s-38.9MiB/s (40.8MB/s-40.8MB/s), io=273MiB (286MB), run=7001-7001msec

$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751642,
  "timestamp_ms" : 1666751642852,
  "time" : "Wed Oct 26 04:34:02 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 955027456,
        "io_kbytes" : 932644,
        "bw_bytes" : 136413006,
        "bw" : 133215,
        "iops" : 266431.366948,
        "runtime" : 7001,
        "total_ios" : 1865286,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1533,
          "max" : 25778,
          "mean" : 2232.692276,
          "stddev" : 689.704731,
          "N" : 1865287
        },
        "clat_ns" : {
          "min" : 231,
          "max" : 102753,
          "mean" : 5046.237951,
          "stddev" : 1921.027764,
          "N" : 1865287,
          "percentile" : {
            "1.000000" : 4048,
            "5.000000" : 4080,
            "10.000000" : 4128,
            "20.000000" : 4128,
            "30.000000" : 4192,
            "40.000000" : 4256,
            "50.000000" : 5216,
            "60.000000" : 5280,
            "70.000000" : 5344,
            "80.000000" : 5408,
            "90.000000" : 5472,
            "95.000000" : 5664,
            "99.000000" : 14400,
            "99.500000" : 18304,
            "99.900000" : 22656,
            "99.950000" : 22912,
            "99.990000" : 61696
          }
        },
        "lat_ns" : {
          "min" : 5651,
          "max" : 104316,
          "mean" : 7278.930227,
          "stddev" : 1850.805954,
          "N" : 1865287
        },
        "bw_min" : 132892,
        "bw_max" : 133392,
        "bw_agg" : 100.000000,
        "bw_mean" : 133246.500000,
        "bw_dev" : 157.042253,
        "bw_samples" : 14,
        "iops_min" : 265784,
        "iops_max" : 266785,
        "iops_mean" : 266493.214286,
        "iops_stddev" : 314.111734,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 35.300000,
      "sys_cpu" : 64.671429,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.024286,
        "10" : 97.589217,
        "20" : 1.995404,
        "50" : 0.367129,
        "100" : 0.021820,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140835: Wed Oct 26 04:34:02 2022
  read: IOPS=266k, BW=130MiB/s (136MB/s)(911MiB/7001msec)
    slat (nsec): min=1533, max=25778, avg=2232.69, stdev=689.70
    clat (nsec): min=231, max=102753, avg=5046.24, stdev=1921.03
     lat (usec): min=5, max=104, avg= 7.28, stdev= 1.85
    clat percentiles (nsec):
     |  1.00th=[ 4048],  5.00th=[ 4080], 10.00th=[ 4128], 20.00th=[ 4128],
     | 30.00th=[ 4192], 40.00th=[ 4256], 50.00th=[ 5216], 60.00th=[ 5280],
     | 70.00th=[ 5344], 80.00th=[ 5408], 90.00th=[ 5472], 95.00th=[ 5664],
     | 99.00th=[14400], 99.50th=[18304], 99.90th=[22656], 99.95th=[22912],
     | 99.99th=[61696]
   bw (  KiB/s): min=132892, max=133392, per=100.00%, avg=133246.50, stdev=157.04, samples=14
   iops        : min=265784, max=266785, avg=266493.21, stdev=314.11, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.02%, 10=97.59%, 20=2.00%, 50=0.37%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=35.30%, sys=64.67%, ctx=23, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1865286,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=911MiB (955MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751657,
  "timestamp_ms" : 1666751657907,
  "time" : "Wed Oct 26 04:34:17 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 949441536,
        "io_kbytes" : 927189,
        "bw_bytes" : 135615131,
        "bw" : 132436,
        "iops" : 264873.018140,
        "runtime" : 7001,
        "total_ios" : 1854376,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1552,
          "max" : 24926,
          "mean" : 2245.204077,
          "stddev" : 687.051828,
          "N" : 1854377
        },
        "clat_ns" : {
          "min" : 230,
          "max" : 103073,
          "mean" : 5066.601525,
          "stddev" : 1903.677678,
          "N" : 1854377,
          "percentile" : {
            "1.000000" : 4080,
            "5.000000" : 4128,
            "10.000000" : 4128,
            "20.000000" : 4192,
            "30.000000" : 4192,
            "40.000000" : 4256,
            "50.000000" : 5216,
            "60.000000" : 5280,
            "70.000000" : 5408,
            "80.000000" : 5472,
            "90.000000" : 5536,
            "95.000000" : 5664,
            "99.000000" : 14400,
            "99.500000" : 18304,
            "99.900000" : 22400,
            "99.950000" : 22912,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5641,
          "max" : 104686,
          "mean" : 7311.805603,
          "stddev" : 1830.883007,
          "N" : 1854377
        },
        "bw_min" : 132278,
        "bw_max" : 133014,
        "bw_agg" : 100.000000,
        "bw_mean" : 132478.384615,
        "bw_dev" : 208.302880,
        "bw_samples" : 13,
        "iops_min" : 264556,
        "iops_max" : 266028,
        "iops_mean" : 264957.000000,
        "iops_stddev" : 416.682533,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 30.785714,
      "sys_cpu" : 69.185714,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.017310,
        "10" : 97.592667,
        "20" : 1.983039,
        "50" : 0.384172,
        "100" : 0.020600,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=140884: Wed Oct 26 04:34:17 2022
  read: IOPS=265k, BW=129MiB/s (136MB/s)(905MiB/7001msec)
    slat (nsec): min=1552, max=24926, avg=2245.20, stdev=687.05
    clat (nsec): min=230, max=103073, avg=5066.60, stdev=1903.68
     lat (usec): min=5, max=104, avg= 7.31, stdev= 1.83
    clat percentiles (nsec):
     |  1.00th=[ 4080],  5.00th=[ 4128], 10.00th=[ 4128], 20.00th=[ 4192],
     | 30.00th=[ 4192], 40.00th=[ 4256], 50.00th=[ 5216], 60.00th=[ 5280],
     | 70.00th=[ 5408], 80.00th=[ 5472], 90.00th=[ 5536], 95.00th=[ 5664],
     | 99.00th=[14400], 99.50th=[18304], 99.90th=[22400], 99.95th=[22912],
     | 99.99th=[61184]
   bw (  KiB/s): min=132278, max=133014, per=100.00%, avg=132478.38, stdev=208.30, samples=13
   iops        : min=264556, max=266028, avg=264957.00, stdev=416.68, samples=13
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.02%, 10=97.59%, 20=1.98%, 50=0.38%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=30.79%, sys=69.19%, ctx=23, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1854376,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=129MiB/s (136MB/s), 129MiB/s-129MiB/s (136MB/s-136MB/s), io=905MiB (949MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751671,
  "timestamp_ms" : 1666751671714,
  "time" : "Wed Oct 26 04:34:31 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 764834304,
        "io_kbytes" : 746908,
        "bw_bytes" : 109246436,
        "bw" : 106685,
        "iops" : 213371.804028,
        "runtime" : 7001,
        "total_ios" : 1493816,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1382,
          "max" : 25729,
          "mean" : 1532.744973,
          "stddev" : 144.972706,
          "N" : 1493816
        },
        "clat_ns" : {
          "min" : 130,
          "max" : 117480,
          "mean" : 7572.991351,
          "stddev" : 2518.779534,
          "N" : 1493817,
          "percentile" : {
            "1.000000" : 5152,
            "5.000000" : 5152,
            "10.000000" : 5216,
            "20.000000" : 5280,
            "30.000000" : 5344,
            "40.000000" : 5472,
            "50.000000" : 9024,
            "60.000000" : 9024,
            "70.000000" : 9152,
            "80.000000" : 9280,
            "90.000000" : 9408,
            "95.000000" : 10432,
            "99.000000" : 17024,
            "99.500000" : 18048,
            "99.900000" : 19840,
            "99.950000" : 23168,
            "99.990000" : 60160
          }
        },
        "lat_ns" : {
          "min" : 6092,
          "max" : 119193,
          "mean" : 9105.736391,
          "stddev" : 2595.087936,
          "N" : 1493817
        },
        "bw_min" : 106453,
        "bw_max" : 106968,
        "bw_agg" : 100.000000,
        "bw_mean" : 106711.785714,
        "bw_dev" : 136.440000,
        "bw_samples" : 14,
        "iops_min" : 212906,
        "iops_max" : 213936,
        "iops_mean" : 213423.428571,
        "iops_stddev" : 272.960328,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.857143,
      "sys_cpu" : 57.628571,
      "ctx" : 747066,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 94.517866,
        "20" : 5.375160,
        "50" : 0.077787,
        "100" : 0.020351,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=140924: Wed Oct 26 04:34:31 2022
  read: IOPS=213k, BW=104MiB/s (109MB/s)(729MiB/7001msec)
    slat (nsec): min=1382, max=25729, avg=1532.74, stdev=144.97
    clat (nsec): min=130, max=117480, avg=7572.99, stdev=2518.78
     lat (usec): min=6, max=119, avg= 9.11, stdev= 2.60
    clat percentiles (nsec):
     |  1.00th=[ 5152],  5.00th=[ 5152], 10.00th=[ 5216], 20.00th=[ 5280],
     | 30.00th=[ 5344], 40.00th=[ 5472], 50.00th=[ 9024], 60.00th=[ 9024],
     | 70.00th=[ 9152], 80.00th=[ 9280], 90.00th=[ 9408], 95.00th=[10432],
     | 99.00th=[17024], 99.50th=[18048], 99.90th=[19840], 99.95th=[23168],
     | 99.99th=[60160]
   bw (  KiB/s): min=106453, max=106968, per=100.00%, avg=106711.79, stdev=136.44, samples=14
   iops        : min=212906, max=213936, avg=213423.43, stdev=272.96, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=94.52%, 20=5.38%, 50=0.08%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=19.86%, sys=57.63%, ctx=747066, majf=0, minf=1
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1493816,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=729MiB (765MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_io_uring-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751685,
  "timestamp_ms" : 1666751685358,
  "time" : "Wed Oct 26 04:34:45 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 765723648,
        "io_kbytes" : 747777,
        "bw_bytes" : 109373467,
        "bw" : 106810,
        "iops" : 213619.911441,
        "runtime" : 7001,
        "total_ios" : 1495553,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1352,
          "max" : 22452,
          "mean" : 1493.274781,
          "stddev" : 148.492491,
          "N" : 1495553
        },
        "clat_ns" : {
          "min" : 130,
          "max" : 122009,
          "mean" : 7620.069698,
          "stddev" : 2548.142265,
          "N" : 1495554,
          "percentile" : {
            "1.000000" : 5152,
            "5.000000" : 5216,
            "10.000000" : 5216,
            "20.000000" : 5344,
            "30.000000" : 5408,
            "40.000000" : 5536,
            "50.000000" : 8896,
            "60.000000" : 9024,
            "70.000000" : 9024,
            "80.000000" : 9152,
            "90.000000" : 9408,
            "95.000000" : 10560,
            "99.000000" : 17280,
            "99.500000" : 18048,
            "99.900000" : 20608,
            "99.950000" : 23168,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 6161,
          "max" : 123401,
          "mean" : 9113.344539,
          "stddev" : 2616.635881,
          "N" : 1495554
        },
        "bw_min" : 106727,
        "bw_max" : 106938,
        "bw_agg" : 100.000000,
        "bw_mean" : 106835.214286,
        "bw_dev" : 72.031167,
        "bw_samples" : 14,
        "iops_min" : 213454,
        "iops_max" : 213876,
        "iops_mean" : 213670.500000,
        "iops_stddev" : 144.097590,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.214286,
      "sys_cpu" : 60.571429,
      "ctx" : 748054,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 91.554763,
        "20" : 8.329695,
        "50" : 0.084517,
        "100" : 0.016984,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=140964: Wed Oct 26 04:34:45 2022
  read: IOPS=214k, BW=104MiB/s (109MB/s)(730MiB/7001msec)
    slat (nsec): min=1352, max=22452, avg=1493.27, stdev=148.49
    clat (nsec): min=130, max=122009, avg=7620.07, stdev=2548.14
     lat (usec): min=6, max=123, avg= 9.11, stdev= 2.62
    clat percentiles (nsec):
     |  1.00th=[ 5152],  5.00th=[ 5216], 10.00th=[ 5216], 20.00th=[ 5344],
     | 30.00th=[ 5408], 40.00th=[ 5536], 50.00th=[ 8896], 60.00th=[ 9024],
     | 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[ 9408], 95.00th=[10560],
     | 99.00th=[17280], 99.50th=[18048], 99.90th=[20608], 99.95th=[23168],
     | 99.99th=[61184]
   bw (  KiB/s): min=106727, max=106938, per=100.00%, avg=106835.21, stdev=72.03, samples=14
   iops        : min=213454, max=213876, avg=213670.50, stdev=144.10, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=91.55%, 20=8.33%, 50=0.08%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=16.21%, sys=60.57%, ctx=748054, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1495553,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=730MiB (766MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt': No such file or directory

$ rm /tmp/fio-output_512_2_io_uring-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751698,
  "timestamp_ms" : 1666751698998,
  "time" : "Wed Oct 26 04:34:58 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 762477568,
        "io_kbytes" : 744607,
        "bw_bytes" : 108909808,
        "bw" : 106357,
        "iops" : 212714.326525,
        "runtime" : 7001,
        "total_ios" : 1489213,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1352,
          "max" : 26169,
          "mean" : 1505.332639,
          "stddev" : 162.198529,
          "N" : 1489213
        },
        "clat_ns" : {
          "min" : 140,
          "max" : 121859,
          "mean" : 7639.105499,
          "stddev" : 2552.195648,
          "N" : 1489214,
          "percentile" : {
            "1.000000" : 5088,
            "5.000000" : 5152,
            "10.000000" : 5216,
            "20.000000" : 5280,
            "30.000000" : 5344,
            "40.000000" : 5600,
            "50.000000" : 8896,
            "60.000000" : 9024,
            "70.000000" : 9024,
            "80.000000" : 9152,
            "90.000000" : 9664,
            "95.000000" : 10560,
            "99.000000" : 17280,
            "99.500000" : 18048,
            "99.900000" : 20608,
            "99.950000" : 23168,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 6061,
          "max" : 123251,
          "mean" : 9144.438204,
          "stddev" : 2629.067758,
          "N" : 1489214
        },
        "bw_min" : 105852,
        "bw_max" : 106636,
        "bw_agg" : 100.000000,
        "bw_mean" : 106381.928571,
        "bw_dev" : 196.151615,
        "bw_samples" : 14,
        "iops_min" : 211705,
        "iops_max" : 213272,
        "iops_mean" : 212763.928571,
        "iops_stddev" : 392.095448,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.128571,
      "sys_cpu" : 59.557143,
      "ctx" : 745014,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 90.268484,
        "20" : 9.615884,
        "50" : 0.085481,
        "100" : 0.017727,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=141004: Wed Oct 26 04:34:58 2022
  read: IOPS=213k, BW=104MiB/s (109MB/s)(727MiB/7001msec)
    slat (nsec): min=1352, max=26169, avg=1505.33, stdev=162.20
    clat (nsec): min=140, max=121859, avg=7639.11, stdev=2552.20
     lat (usec): min=6, max=123, avg= 9.14, stdev= 2.63
    clat percentiles (nsec):
     |  1.00th=[ 5088],  5.00th=[ 5152], 10.00th=[ 5216], 20.00th=[ 5280],
     | 30.00th=[ 5344], 40.00th=[ 5600], 50.00th=[ 8896], 60.00th=[ 9024],
     | 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[ 9664], 95.00th=[10560],
     | 99.00th=[17280], 99.50th=[18048], 99.90th=[20608], 99.95th=[23168],
     | 99.99th=[62720]
   bw (  KiB/s): min=105852, max=106636, per=100.00%, avg=106381.93, stdev=196.15, samples=14
   iops        : min=211705, max=213272, avg=212763.93, stdev=392.10, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=90.27%, 20=9.62%, 50=0.09%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=17.13%, sys=59.56%, ctx=745014, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1489213,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=104MiB/s (109MB/s), 104MiB/s-104MiB/s (109MB/s-109MB/s), io=727MiB (762MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-reference_0.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751712,
  "timestamp_ms" : 1666751712628,
  "time" : "Wed Oct 26 04:35:12 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 779892224,
        "io_kbytes" : 761613,
        "bw_bytes" : 111397260,
        "bw" : 108786,
        "iops" : 217572.489644,
        "runtime" : 7001,
        "total_ios" : 1523225,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1392,
          "max" : 26299,
          "mean" : 1533.941948,
          "stddev" : 145.787725,
          "N" : 1523226
        },
        "clat_ns" : {
          "min" : 401,
          "max" : 140834,
          "mean" : 7415.475688,
          "stddev" : 1799.292095,
          "N" : 1523226,
          "percentile" : {
            "1.000000" : 6880,
            "5.000000" : 6944,
            "10.000000" : 6944,
            "20.000000" : 7008,
            "30.000000" : 7072,
            "40.000000" : 7072,
            "50.000000" : 7136,
            "60.000000" : 7200,
            "70.000000" : 7264,
            "80.000000" : 7328,
            "90.000000" : 7392,
            "95.000000" : 7648,
            "99.000000" : 15808,
            "99.500000" : 19328,
            "99.900000" : 21120,
            "99.950000" : 25472,
            "99.990000" : 64768
          }
        },
        "lat_ns" : {
          "min" : 5911,
          "max" : 142467,
          "mean" : 8949.417635,
          "stddev" : 1803.451576,
          "N" : 1523226
        },
        "bw_min" : 108527,
        "bw_max" : 109183,
        "bw_agg" : 100.000000,
        "bw_mean" : 108831.785714,
        "bw_dev" : 172.703471,
        "bw_samples" : 14,
        "iops_min" : 217054,
        "iops_max" : 218366,
        "iops_mean" : 217663.714286,
        "iops_stddev" : 345.357881,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 23.371429,
      "sys_cpu" : 54.428571,
      "ctx" : 761236,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 97.033400,
        "20" : 2.567119,
        "50" : 0.367510,
        "100" : 0.019761,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2179709,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 10715,
      "write_ticks" : 0,
      "in_queue" : 10715,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=141045: Wed Oct 26 04:35:12 2022
  read: IOPS=218k, BW=106MiB/s (111MB/s)(744MiB/7001msec)
    slat (nsec): min=1392, max=26299, avg=1533.94, stdev=145.79
    clat (nsec): min=401, max=140834, avg=7415.48, stdev=1799.29
     lat (usec): min=5, max=142, avg= 8.95, stdev= 1.80
    clat percentiles (nsec):
     |  1.00th=[ 6880],  5.00th=[ 6944], 10.00th=[ 6944], 20.00th=[ 7008],
     | 30.00th=[ 7072], 40.00th=[ 7072], 50.00th=[ 7136], 60.00th=[ 7200],
     | 70.00th=[ 7264], 80.00th=[ 7328], 90.00th=[ 7392], 95.00th=[ 7648],
     | 99.00th=[15808], 99.50th=[19328], 99.90th=[21120], 99.95th=[25472],
     | 99.99th=[64768]
   bw (  KiB/s): min=108527, max=109183, per=100.00%, avg=108831.79, stdev=172.70, samples=14
   iops        : min=217054, max=218366, avg=217663.71, stdev=345.36, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=97.03%, 20=2.57%, 50=0.37%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=23.37%, sys=54.43%, ctx=761236, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1523225,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s), io=744MiB (780MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2179709/0, merge=0/0, ticks=10715/0, in_queue=10715, util=99.09%

$ rm /tmp/fio-output_512_2_libaio-reference_1.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751726,
  "timestamp_ms" : 1666751726267,
  "time" : "Wed Oct 26 04:35:26 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 770420224,
        "io_kbytes" : 752363,
        "bw_bytes" : 110044311,
        "bw" : 107465,
        "iops" : 214930.009999,
        "runtime" : 7001,
        "total_ios" : 1504725,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1412,
          "max" : 15820,
          "mean" : 1557.443289,
          "stddev" : 138.168896,
          "N" : 1504726
        },
        "clat_ns" : {
          "min" : 400,
          "max" : 123452,
          "mean" : 7487.520184,
          "stddev" : 1736.521261,
          "N" : 1504726,
          "percentile" : {
            "1.000000" : 6944,
            "5.000000" : 7008,
            "10.000000" : 7008,
            "20.000000" : 7072,
            "30.000000" : 7136,
            "40.000000" : 7200,
            "50.000000" : 7264,
            "60.000000" : 7264,
            "70.000000" : 7328,
            "80.000000" : 7392,
            "90.000000" : 7456,
            "95.000000" : 7712,
            "99.000000" : 15680,
            "99.500000" : 19328,
            "99.900000" : 20608,
            "99.950000" : 24448,
            "99.990000" : 64256
          }
        },
        "lat_ns" : {
          "min" : 5961,
          "max" : 125115,
          "mean" : 9044.963474,
          "stddev" : 1739.023859,
          "N" : 1504726
        },
        "bw_min" : 107304,
        "bw_max" : 107838,
        "bw_agg" : 100.000000,
        "bw_mean" : 107519.384615,
        "bw_dev" : 142.703970,
        "bw_samples" : 13,
        "iops_min" : 214608,
        "iops_max" : 215674,
        "iops_mean" : 215038.615385,
        "iops_stddev" : 285.036120,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 23.071429,
      "sys_cpu" : 55.214286,
      "ctx" : 751974,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.981841,
        "20" : 2.622007,
        "50" : 0.365117,
        "100" : 0.019671,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2102056,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 10370,
      "write_ticks" : 0,
      "in_queue" : 10370,
      "util" : 99.085923
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=141084: Wed Oct 26 04:35:26 2022
  read: IOPS=215k, BW=105MiB/s (110MB/s)(735MiB/7001msec)
    slat (nsec): min=1412, max=15820, avg=1557.44, stdev=138.17
    clat (nsec): min=400, max=123452, avg=7487.52, stdev=1736.52
     lat (usec): min=5, max=125, avg= 9.04, stdev= 1.74
    clat percentiles (nsec):
     |  1.00th=[ 6944],  5.00th=[ 7008], 10.00th=[ 7008], 20.00th=[ 7072],
     | 30.00th=[ 7136], 40.00th=[ 7200], 50.00th=[ 7264], 60.00th=[ 7264],
     | 70.00th=[ 7328], 80.00th=[ 7392], 90.00th=[ 7456], 95.00th=[ 7712],
     | 99.00th=[15680], 99.50th=[19328], 99.90th=[20608], 99.95th=[24448],
     | 99.99th=[64256]
   bw (  KiB/s): min=107304, max=107838, per=100.00%, avg=107519.38, stdev=142.70, samples=13
   iops        : min=214608, max=215674, avg=215038.62, stdev=285.04, samples=13
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=96.98%, 20=2.62%, 50=0.37%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=23.07%, sys=55.21%, ctx=751974, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1504725,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=735MiB (770MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2102056/0, merge=0/0, ticks=10370/0, in_queue=10370, util=99.09%

$ rm /tmp/fio-output_512_2_libaio-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751739,
  "timestamp_ms" : 1666751739922,
  "time" : "Wed Oct 26 04:35:39 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 771123712,
        "io_kbytes" : 753050,
        "bw_bytes" : 110144795,
        "bw" : 107563,
        "iops" : 215126.410513,
        "runtime" : 7001,
        "total_ios" : 1506100,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1392,
          "max" : 26369,
          "mean" : 1556.186424,
          "stddev" : 150.779054,
          "N" : 1506100
        },
        "clat_ns" : {
          "min" : 420,
          "max" : 149550,
          "mean" : 7484.908665,
          "stddev" : 1738.332903,
          "N" : 1506101,
          "percentile" : {
            "1.000000" : 6944,
            "5.000000" : 6944,
            "10.000000" : 7008,
            "20.000000" : 7072,
            "30.000000" : 7136,
            "40.000000" : 7200,
            "50.000000" : 7264,
            "60.000000" : 7264,
            "70.000000" : 7328,
            "80.000000" : 7392,
            "90.000000" : 7520,
            "95.000000" : 7776,
            "99.000000" : 15680,
            "99.500000" : 18816,
            "99.900000" : 20864,
            "99.950000" : 24960,
            "99.990000" : 62208
          }
        },
        "lat_ns" : {
          "min" : 5911,
          "max" : 151223,
          "mean" : 9041.095007,
          "stddev" : 1740.860134,
          "N" : 1506101
        },
        "bw_min" : 107410,
        "bw_max" : 107798,
        "bw_agg" : 100.000000,
        "bw_mean" : 107620.692308,
        "bw_dev" : 114.054362,
        "bw_samples" : 13,
        "iops_min" : 214820,
        "iops_max" : 215596,
        "iops_mean" : 215241.230769,
        "iops_stddev" : 227.980319,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 25.185714,
      "sys_cpu" : 52.814286,
      "ctx" : 752769,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.987119,
        "20" : 2.608924,
        "50" : 0.373880,
        "100" : 0.017197,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2105479,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 10382,
      "write_ticks" : 0,
      "in_queue" : 10382,
      "util" : 99.085923
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=141122: Wed Oct 26 04:35:39 2022
  read: IOPS=215k, BW=105MiB/s (110MB/s)(735MiB/7001msec)
    slat (nsec): min=1392, max=26369, avg=1556.19, stdev=150.78
    clat (nsec): min=420, max=149550, avg=7484.91, stdev=1738.33
     lat (usec): min=5, max=151, avg= 9.04, stdev= 1.74
    clat percentiles (nsec):
     |  1.00th=[ 6944],  5.00th=[ 6944], 10.00th=[ 7008], 20.00th=[ 7072],
     | 30.00th=[ 7136], 40.00th=[ 7200], 50.00th=[ 7264], 60.00th=[ 7264],
     | 70.00th=[ 7328], 80.00th=[ 7392], 90.00th=[ 7520], 95.00th=[ 7776],
     | 99.00th=[15680], 99.50th=[18816], 99.90th=[20864], 99.95th=[24960],
     | 99.99th=[62208]
   bw (  KiB/s): min=107410, max=107798, per=100.00%, avg=107620.69, stdev=114.05, samples=13
   iops        : min=214820, max=215596, avg=215241.23, stdev=227.98, samples=13
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=96.99%, 20=2.61%, 50=0.37%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=25.19%, sys=52.81%, ctx=752769, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1506100,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=735MiB (771MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2105479/0, merge=0/0, ticks=10382/0, in_queue=10382, util=99.09%

$ rm /tmp/fio-output_512_2_libaio-bdev_aio_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751754,
  "timestamp_ms" : 1666751754974,
  "time" : "Wed Oct 26 04:35:54 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 925439488,
        "io_kbytes" : 903749,
        "bw_bytes" : 132186757,
        "bw" : 129088,
        "iops" : 258177.117555,
        "runtime" : 7001,
        "total_ios" : 1807498,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1603,
          "max" : 30868,
          "mean" : 2315.003258,
          "stddev" : 734.458639,
          "N" : 1807498
        },
        "clat_ns" : {
          "min" : 220,
          "max" : 153418,
          "mean" : 5163.300420,
          "stddev" : 1884.570066,
          "N" : 1807499,
          "percentile" : {
            "1.000000" : 4128,
            "5.000000" : 4128,
            "10.000000" : 4192,
            "20.000000" : 4192,
            "30.000000" : 4256,
            "40.000000" : 4384,
            "50.000000" : 5344,
            "60.000000" : 5408,
            "70.000000" : 5536,
            "80.000000" : 5600,
            "90.000000" : 5792,
            "95.000000" : 5920,
            "99.000000" : 13376,
            "99.500000" : 18048,
            "99.900000" : 20608,
            "99.950000" : 20864,
            "99.990000" : 63744
          }
        },
        "lat_ns" : {
          "min" : 5731,
          "max" : 155091,
          "mean" : 7478.303306,
          "stddev" : 1803.617973,
          "N" : 1807499
        },
        "bw_min" : 128902,
        "bw_max" : 129385,
        "bw_agg" : 100.000000,
        "bw_mean" : 129123.500000,
        "bw_dev" : 156.196006,
        "bw_samples" : 14,
        "iops_min" : 257804,
        "iops_max" : 258770,
        "iops_mean" : 258247.000000,
        "iops_stddev" : 312.392012,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 57.414286,
      "sys_cpu" : 42.557143,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 1.656212,
        "2" : 98.343788,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.012172,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 97.917453,
        "20" : 1.799283,
        "50" : 0.244924,
        "100" : 0.017317,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=141171: Wed Oct 26 04:35:54 2022
  read: IOPS=258k, BW=126MiB/s (132MB/s)(883MiB/7001msec)
    slat (nsec): min=1603, max=30868, avg=2315.00, stdev=734.46
    clat (nsec): min=220, max=153418, avg=5163.30, stdev=1884.57
     lat (usec): min=5, max=155, avg= 7.48, stdev= 1.80
    clat percentiles (nsec):
     |  1.00th=[ 4128],  5.00th=[ 4128], 10.00th=[ 4192], 20.00th=[ 4192],
     | 30.00th=[ 4256], 40.00th=[ 4384], 50.00th=[ 5344], 60.00th=[ 5408],
     | 70.00th=[ 5536], 80.00th=[ 5600], 90.00th=[ 5792], 95.00th=[ 5920],
     | 99.00th=[13376], 99.50th=[18048], 99.90th=[20608], 99.95th=[20864],
     | 99.99th=[63744]
   bw (  KiB/s): min=128902, max=129385, per=100.00%, avg=129123.50, stdev=156.20, samples=14
   iops        : min=257804, max=258770, avg=258247.00, stdev=312.39, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=97.92%, 20=1.80%, 50=0.24%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=57.41%, sys=42.56%, ctx=23, majf=0, minf=0
  IO depths    : 1=1.7%, 2=98.3%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1807498,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=883MiB (925MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-bdev_aio_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751770,
  "timestamp_ms" : 1666751770080,
  "time" : "Wed Oct 26 04:36:10 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 910187008,
        "io_kbytes" : 888854,
        "bw_bytes" : 130008142,
        "bw" : 126961,
        "iops" : 253921.868305,
        "runtime" : 7001,
        "total_ios" : 1777707,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1593,
          "max" : 30537,
          "mean" : 2352.376895,
          "stddev" : 772.406286,
          "N" : 1777708
        },
        "clat_ns" : {
          "min" : 241,
          "max" : 135454,
          "mean" : 5251.168977,
          "stddev" : 1853.294464,
          "N" : 1777707,
          "percentile" : {
            "1.000000" : 4192,
            "5.000000" : 4192,
            "10.000000" : 4256,
            "20.000000" : 4256,
            "30.000000" : 4320,
            "40.000000" : 4448,
            "50.000000" : 5472,
            "60.000000" : 5536,
            "70.000000" : 5664,
            "80.000000" : 5728,
            "90.000000" : 5920,
            "95.000000" : 5984,
            "99.000000" : 13376,
            "99.500000" : 17792,
            "99.900000" : 20608,
            "99.950000" : 20864,
            "99.990000" : 60160
          }
        },
        "lat_ns" : {
          "min" : 5781,
          "max" : 137148,
          "mean" : 7603.546254,
          "stddev" : 1761.177099,
          "N" : 1777707
        },
        "bw_min" : 126711,
        "bw_max" : 127293,
        "bw_agg" : 100.000000,
        "bw_mean" : 126994.142857,
        "bw_dev" : 203.450942,
        "bw_samples" : 14,
        "iops_min" : 253422,
        "iops_max" : 254586,
        "iops_mean" : 253988.357143,
        "iops_stddev" : 407.005879,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 58.271429,
      "sys_cpu" : 41.700000,
      "ctx" : 20,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 1.653985,
        "2" : 98.346015,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.014457,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 97.943981,
        "20" : 1.808003,
        "50" : 0.211565,
        "100" : 0.018394,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=141222: Wed Oct 26 04:36:10 2022
  read: IOPS=254k, BW=124MiB/s (130MB/s)(868MiB/7001msec)
    slat (nsec): min=1593, max=30537, avg=2352.38, stdev=772.41
    clat (nsec): min=241, max=135454, avg=5251.17, stdev=1853.29
     lat (usec): min=5, max=137, avg= 7.60, stdev= 1.76
    clat percentiles (nsec):
     |  1.00th=[ 4192],  5.00th=[ 4192], 10.00th=[ 4256], 20.00th=[ 4256],
     | 30.00th=[ 4320], 40.00th=[ 4448], 50.00th=[ 5472], 60.00th=[ 5536],
     | 70.00th=[ 5664], 80.00th=[ 5728], 90.00th=[ 5920], 95.00th=[ 5984],
     | 99.00th=[13376], 99.50th=[17792], 99.90th=[20608], 99.95th=[20864],
     | 99.99th=[60160]
   bw (  KiB/s): min=126711, max=127293, per=100.00%, avg=126994.14, stdev=203.45, samples=14
   iops        : min=253422, max=254586, avg=253988.36, stdev=407.01, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=97.94%, 20=1.81%, 50=0.21%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=58.27%, sys=41.70%, ctx=20, majf=0, minf=0
  IO depths    : 1=1.7%, 2=98.3%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1777707,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=868MiB (910MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-bdev_aio_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751008,
  "timestamp_ms" : 1666751008324,
  "time" : "Wed Oct 26 04:23:28 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 291316224,
        "io_kbytes" : 284488,
        "bw_bytes" : 41610659,
        "bw" : 40635,
        "iops" : 81270.675618,
        "runtime" : 7001,
        "total_ios" : 568976,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2325,
          "max" : 28083,
          "mean" : 2529.571946,
          "stddev" : 230.071012,
          "N" : 568977
        },
        "clat_ns" : {
          "min" : 270,
          "max" : 111159,
          "mean" : 9106.443340,
          "stddev" : 1251.821492,
          "N" : 568976,
          "percentile" : {
            "1.000000" : 8768,
            "5.000000" : 8768,
            "10.000000" : 8768,
            "20.000000" : 8896,
            "30.000000" : 8896,
            "40.000000" : 8896,
            "50.000000" : 8896,
            "60.000000" : 8896,
            "70.000000" : 9024,
            "80.000000" : 9024,
            "90.000000" : 9024,
            "95.000000" : 9280,
            "99.000000" : 15552,
            "99.500000" : 16512,
            "99.900000" : 19072,
            "99.950000" : 21632,
            "99.990000" : 55552
          }
        },
        "lat_ns" : {
          "min" : 8857,
          "max" : 113743,
          "mean" : 11636.015347,
          "stddev" : 1270.086719,
          "N" : 568976
        },
        "bw_min" : 40559,
        "bw_max" : 40758,
        "bw_agg" : 100.000000,
        "bw_mean" : 40646.000000,
        "bw_dev" : 70.886692,
        "bw_samples" : 14,
        "iops_min" : 81118,
        "iops_max" : 81516,
        "iops_mean" : 81292.000000,
        "iops_stddev" : 141.773384,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.200000,
      "sys_cpu" : 43.085714,
      "ctx" : 568842,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.012479,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 97.371067,
        "20" : 2.537365,
        "50" : 0.064678,
        "100" : 0.014236,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=138867: Wed Oct 26 04:23:28 2022
  read: IOPS=81.3k, BW=39.7MiB/s (41.6MB/s)(278MiB/7001msec)
    slat (nsec): min=2325, max=28083, avg=2529.57, stdev=230.07
    clat (nsec): min=270, max=111159, avg=9106.44, stdev=1251.82
     lat (usec): min=8, max=113, avg=11.64, stdev= 1.27
    clat percentiles (nsec):
     |  1.00th=[ 8768],  5.00th=[ 8768], 10.00th=[ 8768], 20.00th=[ 8896],
     | 30.00th=[ 8896], 40.00th=[ 8896], 50.00th=[ 8896], 60.00th=[ 8896],
     | 70.00th=[ 9024], 80.00th=[ 9024], 90.00th=[ 9024], 95.00th=[ 9280],
     | 99.00th=[15552], 99.50th=[16512], 99.90th=[19072], 99.95th=[21632],
     | 99.99th=[55552]
   bw (  KiB/s): min=40559, max=40758, per=100.00%, avg=40646.00, stdev=70.89, samples=14
   iops        : min=81118, max=81516, avg=81292.00, stdev=141.77, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=97.37%, 20=2.54%, 50=0.06%, 100=0.01%, 250=0.01%
  cpu          : usr=16.20%, sys=43.09%, ctx=568842, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=568976,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=39.7MiB/s (41.6MB/s), 39.7MiB/s-39.7MiB/s (41.6MB/s-41.6MB/s), io=278MiB (291MB), run=7001-7001msec

$ cat /tmp/fio-output_512_2_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751785,
  "timestamp_ms" : 1666751785122,
  "time" : "Wed Oct 26 04:36:25 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 922618368,
        "io_kbytes" : 900994,
        "bw_bytes" : 131783797,
        "bw" : 128695,
        "iops" : 257389.944294,
        "runtime" : 7001,
        "total_ios" : 1801987,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1583,
          "max" : 29826,
          "mean" : 2299.051626,
          "stddev" : 743.697493,
          "N" : 1801988
        },
        "clat_ns" : {
          "min" : 230,
          "max" : 138339,
          "mean" : 5203.031010,
          "stddev" : 1872.544552,
          "N" : 1801987,
          "percentile" : {
            "1.000000" : 4128,
            "5.000000" : 4192,
            "10.000000" : 4192,
            "20.000000" : 4256,
            "30.000000" : 4320,
            "40.000000" : 4448,
            "50.000000" : 5408,
            "60.000000" : 5472,
            "70.000000" : 5600,
            "80.000000" : 5664,
            "90.000000" : 5856,
            "95.000000" : 5984,
            "99.000000" : 13376,
            "99.500000" : 18048,
            "99.900000" : 20864,
            "99.950000" : 21120,
            "99.990000" : 61184
          }
        },
        "lat_ns" : {
          "min" : 5740,
          "max" : 140022,
          "mean" : 7502.082299,
          "stddev" : 1785.297905,
          "N" : 1801987
        },
        "bw_min" : 128549,
        "bw_max" : 129325,
        "bw_agg" : 100.000000,
        "bw_mean" : 128728.785714,
        "bw_dev" : 240.665161,
        "bw_samples" : 14,
        "iops_min" : 257098,
        "iops_max" : 258650,
        "iops_mean" : 257457.714286,
        "iops_stddev" : 481.220792,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 59.585714,
      "sys_cpu" : 40.385714,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 1.640911,
        "2" : 98.359089,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.011876,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 97.912249,
        "20" : 1.800734,
        "50" : 0.250335,
        "100" : 0.017148,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=141271: Wed Oct 26 04:36:25 2022
  read: IOPS=257k, BW=126MiB/s (132MB/s)(880MiB/7001msec)
    slat (nsec): min=1583, max=29826, avg=2299.05, stdev=743.70
    clat (nsec): min=230, max=138339, avg=5203.03, stdev=1872.54
     lat (usec): min=5, max=140, avg= 7.50, stdev= 1.79
    clat percentiles (nsec):
     |  1.00th=[ 4128],  5.00th=[ 4192], 10.00th=[ 4192], 20.00th=[ 4256],
     | 30.00th=[ 4320], 40.00th=[ 4448], 50.00th=[ 5408], 60.00th=[ 5472],
     | 70.00th=[ 5600], 80.00th=[ 5664], 90.00th=[ 5856], 95.00th=[ 5984],
     | 99.00th=[13376], 99.50th=[18048], 99.90th=[20864], 99.95th=[21120],
     | 99.99th=[61184]
   bw (  KiB/s): min=128549, max=129325, per=100.00%, avg=128728.79, stdev=240.67, samples=14
   iops        : min=257098, max=258650, avg=257457.71, stdev=481.22, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=97.91%, 20=1.80%, 50=0.25%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=59.59%, sys=40.39%, ctx=23, majf=0, minf=0
  IO depths    : 1=1.6%, 2=98.4%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1801987,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=880MiB (923MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751800,
  "timestamp_ms" : 1666751800261,
  "time" : "Wed Oct 26 04:36:40 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 892137472,
        "io_kbytes" : 871228,
        "bw_bytes" : 127430005,
        "bw" : 124443,
        "iops" : 248886.444794,
        "runtime" : 7001,
        "total_ios" : 1742454,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1593,
          "max" : 19317,
          "mean" : 2306.401335,
          "stddev" : 734.829566,
          "N" : 1742455
        },
        "clat_ns" : {
          "min" : 591,
          "max" : 121829,
          "mean" : 5472.362839,
          "stddev" : 1813.157116,
          "N" : 1742454,
          "percentile" : {
            "1.000000" : 4448,
            "5.000000" : 4512,
            "10.000000" : 4512,
            "20.000000" : 4576,
            "30.000000" : 4576,
            "40.000000" : 4640,
            "50.000000" : 5728,
            "60.000000" : 5792,
            "70.000000" : 5856,
            "80.000000" : 5920,
            "90.000000" : 5984,
            "95.000000" : 6176,
            "99.000000" : 13248,
            "99.500000" : 17536,
            "99.900000" : 20352,
            "99.950000" : 20864,
            "99.990000" : 62208
          }
        },
        "lat_ns" : {
          "min" : 5901,
          "max" : 123492,
          "mean" : 7778.763796,
          "stddev" : 1710.924115,
          "N" : 1742454
        },
        "bw_min" : 124145,
        "bw_max" : 124816,
        "bw_agg" : 100.000000,
        "bw_mean" : 124484.384615,
        "bw_dev" : 207.810626,
        "bw_samples" : 13,
        "iops_min" : 248290,
        "iops_max" : 249632,
        "iops_mean" : 248968.692308,
        "iops_stddev" : 415.787483,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.300000,
      "sys_cpu" : 60.671429,
      "ctx" : 18,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 1.483425,
        "2" : 98.516575,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.013200,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 98.015098,
        "20" : 1.779043,
        "50" : 0.169072,
        "100" : 0.019513,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141320: Wed Oct 26 04:36:40 2022
  read: IOPS=249k, BW=122MiB/s (127MB/s)(851MiB/7001msec)
    slat (nsec): min=1593, max=19317, avg=2306.40, stdev=734.83
    clat (nsec): min=591, max=121829, avg=5472.36, stdev=1813.16
     lat (usec): min=5, max=123, avg= 7.78, stdev= 1.71
    clat percentiles (nsec):
     |  1.00th=[ 4448],  5.00th=[ 4512], 10.00th=[ 4512], 20.00th=[ 4576],
     | 30.00th=[ 4576], 40.00th=[ 4640], 50.00th=[ 5728], 60.00th=[ 5792],
     | 70.00th=[ 5856], 80.00th=[ 5920], 90.00th=[ 5984], 95.00th=[ 6176],
     | 99.00th=[13248], 99.50th=[17536], 99.90th=[20352], 99.95th=[20864],
     | 99.99th=[62208]
   bw (  KiB/s): min=124145, max=124816, per=100.00%, avg=124484.38, stdev=207.81, samples=13
   iops        : min=248290, max=249632, avg=248968.69, stdev=415.79, samples=13
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=98.02%, 20=1.78%, 50=0.17%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=39.30%, sys=60.67%, ctx=18, majf=0, minf=0
  IO depths    : 1=1.5%, 2=98.5%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1742454,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=122MiB/s (127MB/s), 122MiB/s-122MiB/s (127MB/s-127MB/s), io=851MiB (892MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751815,
  "timestamp_ms" : 1666751815448,
  "time" : "Wed Oct 26 04:36:55 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 888960000,
        "io_kbytes" : 868125,
        "bw_bytes" : 126976146,
        "bw" : 124000,
        "iops" : 248000.142837,
        "runtime" : 7001,
        "total_ios" : 1736249,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1613,
          "max" : 33963,
          "mean" : 2325.591072,
          "stddev" : 759.135311,
          "N" : 1736249
        },
        "clat_ns" : {
          "min" : 621,
          "max" : 121107,
          "mean" : 5472.580111,
          "stddev" : 1812.845915,
          "N" : 1736250,
          "percentile" : {
            "1.000000" : 4256,
            "5.000000" : 4320,
            "10.000000" : 4512,
            "20.000000" : 4576,
            "30.000000" : 4640,
            "40.000000" : 4704,
            "50.000000" : 5472,
            "60.000000" : 5856,
            "70.000000" : 5920,
            "80.000000" : 5984,
            "90.000000" : 6048,
            "95.000000" : 6240,
            "99.000000" : 13376,
            "99.500000" : 17280,
            "99.900000" : 20096,
            "99.950000" : 20608,
            "99.990000" : 60672
          }
        },
        "lat_ns" : {
          "min" : 5891,
          "max" : 122811,
          "mean" : 7798.170796,
          "stddev" : 1715.249598,
          "N" : 1736250
        },
        "bw_min" : 123725,
        "bw_max" : 124153,
        "bw_agg" : 100.000000,
        "bw_mean" : 124029.571429,
        "bw_dev" : 106.987213,
        "bw_samples" : 14,
        "iops_min" : 247450,
        "iops_max" : 248306,
        "iops_mean" : 248059.000000,
        "iops_stddev" : 214.096671,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 35.585714,
      "sys_cpu" : 64.385714,
      "ctx" : 24,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 1.473003,
        "2" : 98.526997,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.012268,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 97.911532,
        "20" : 1.955998,
        "50" : 0.096991,
        "100" : 0.019410,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141369: Wed Oct 26 04:36:55 2022
  read: IOPS=248k, BW=121MiB/s (127MB/s)(848MiB/7001msec)
    slat (nsec): min=1613, max=33963, avg=2325.59, stdev=759.14
    clat (nsec): min=621, max=121107, avg=5472.58, stdev=1812.85
     lat (usec): min=5, max=122, avg= 7.80, stdev= 1.72
    clat percentiles (nsec):
     |  1.00th=[ 4256],  5.00th=[ 4320], 10.00th=[ 4512], 20.00th=[ 4576],
     | 30.00th=[ 4640], 40.00th=[ 4704], 50.00th=[ 5472], 60.00th=[ 5856],
     | 70.00th=[ 5920], 80.00th=[ 5984], 90.00th=[ 6048], 95.00th=[ 6240],
     | 99.00th=[13376], 99.50th=[17280], 99.90th=[20096], 99.95th=[20608],
     | 99.99th=[60672]
   bw (  KiB/s): min=123725, max=124153, per=100.00%, avg=124029.57, stdev=106.99, samples=14
   iops        : min=247450, max=248306, avg=248059.00, stdev=214.10, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=97.91%, 20=1.96%, 50=0.10%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=35.59%, sys=64.39%, ctx=24, majf=0, minf=0
  IO depths    : 1=1.5%, 2=98.5%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1736249,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=121MiB/s (127MB/s), 121MiB/s-121MiB/s (127MB/s-127MB/s), io=848MiB (889MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_2_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751830,
  "timestamp_ms" : 1666751830581,
  "time" : "Wed Oct 26 04:37:10 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 886879232,
        "io_kbytes" : 866093,
        "bw_bytes" : 126678936,
        "bw" : 123709,
        "iops" : 247419.511498,
        "runtime" : 7001,
        "total_ios" : 1732184,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1623,
          "max" : 26189,
          "mean" : 2344.885200,
          "stddev" : 746.431565,
          "N" : 1732185
        },
        "clat_ns" : {
          "min" : 621,
          "max" : 175389,
          "mean" : 5470.346042,
          "stddev" : 1841.692980,
          "N" : 1732184,
          "percentile" : {
            "1.000000" : 4256,
            "5.000000" : 4320,
            "10.000000" : 4512,
            "20.000000" : 4576,
            "30.000000" : 4640,
            "40.000000" : 4704,
            "50.000000" : 5536,
            "60.000000" : 5856,
            "70.000000" : 5920,
            "80.000000" : 5984,
            "90.000000" : 6048,
            "95.000000" : 6240,
            "99.000000" : 13120,
            "99.500000" : 17024,
            "99.900000" : 20096,
            "99.950000" : 20352,
            "99.990000" : 64768
          }
        },
        "lat_ns" : {
          "min" : 5921,
          "max" : 177122,
          "mean" : 7815.230866,
          "stddev" : 1740.065131,
          "N" : 1732184
        },
        "bw_min" : 123454,
        "bw_max" : 124192,
        "bw_agg" : 100.000000,
        "bw_mean" : 123727.230769,
        "bw_dev" : 248.715753,
        "bw_samples" : 13,
        "iops_min" : 246908,
        "iops_max" : 248384,
        "iops_mean" : 247454.461538,
        "iops_stddev" : 497.431505,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 37.471429,
      "sys_cpu" : 62.500000,
      "ctx" : 22,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 1.468666,
        "2" : 98.531334,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010045,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 98.048995,
        "20" : 1.830464,
        "50" : 0.086019,
        "100" : 0.019109,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141419: Wed Oct 26 04:37:10 2022
  read: IOPS=247k, BW=121MiB/s (127MB/s)(846MiB/7001msec)
    slat (nsec): min=1623, max=26189, avg=2344.89, stdev=746.43
    clat (nsec): min=621, max=175389, avg=5470.35, stdev=1841.69
     lat (usec): min=5, max=177, avg= 7.82, stdev= 1.74
    clat percentiles (nsec):
     |  1.00th=[ 4256],  5.00th=[ 4320], 10.00th=[ 4512], 20.00th=[ 4576],
     | 30.00th=[ 4640], 40.00th=[ 4704], 50.00th=[ 5536], 60.00th=[ 5856],
     | 70.00th=[ 5920], 80.00th=[ 5984], 90.00th=[ 6048], 95.00th=[ 6240],
     | 99.00th=[13120], 99.50th=[17024], 99.90th=[20096], 99.95th=[20352],
     | 99.99th=[64768]
   bw (  KiB/s): min=123454, max=124192, per=100.00%, avg=123727.23, stdev=248.72, samples=13
   iops        : min=246908, max=248384, avg=247454.46, stdev=497.43, samples=13
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=98.05%, 20=1.83%, 50=0.09%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=37.47%, sys=62.50%, ctx=22, majf=0, minf=0
  IO depths    : 1=1.5%, 2=98.5%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1732184,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=121MiB/s (127MB/s), 121MiB/s-121MiB/s (127MB/s-127MB/s), io=846MiB (887MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt
$ rm /tmp/fio-output_512_2_libaio-xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751844,
  "timestamp_ms" : 1666751844361,
  "time" : "Wed Oct 26 04:37:24 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 727684608,
        "io_kbytes" : 710629,
        "bw_bytes" : 103940095,
        "bw" : 101503,
        "iops" : 203007.713184,
        "runtime" : 7001,
        "total_ios" : 1421257,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1472,
          "max" : 38092,
          "mean" : 1673.767642,
          "stddev" : 256.286057,
          "N" : 1421258
        },
        "clat_ns" : {
          "min" : 471,
          "max" : 160200,
          "mean" : 7924.788124,
          "stddev" : 1919.052306,
          "N" : 1421258,
          "percentile" : {
            "1.000000" : 4704,
            "5.000000" : 7328,
            "10.000000" : 7520,
            "20.000000" : 7584,
            "30.000000" : 7648,
            "40.000000" : 7712,
            "50.000000" : 7776,
            "60.000000" : 7840,
            "70.000000" : 7904,
            "80.000000" : 7968,
            "90.000000" : 8032,
            "95.000000" : 8160,
            "99.000000" : 15552,
            "99.500000" : 19584,
            "99.900000" : 25216,
            "99.950000" : 25472,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5992,
          "max" : 162054,
          "mean" : 9598.555765,
          "stddev" : 1912.374899,
          "N" : 1421258
        },
        "bw_min" : 101229,
        "bw_max" : 101973,
        "bw_agg" : 100.000000,
        "bw_mean" : 101532.571429,
        "bw_dev" : 208.463136,
        "bw_samples" : 14,
        "iops_min" : 202458,
        "iops_max" : 203946,
        "iops_mean" : 203065.214286,
        "iops_stddev" : 416.847542,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 22.600000,
      "sys_cpu" : 57.742857,
      "ctx" : 681640,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.858907,
        "20" : 2.760233,
        "50" : 0.346454,
        "100" : 0.023078,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=141458: Wed Oct 26 04:37:24 2022
  read: IOPS=203k, BW=99.1MiB/s (104MB/s)(694MiB/7001msec)
    slat (nsec): min=1472, max=38092, avg=1673.77, stdev=256.29
    clat (nsec): min=471, max=160200, avg=7924.79, stdev=1919.05
     lat (usec): min=5, max=162, avg= 9.60, stdev= 1.91
    clat percentiles (nsec):
     |  1.00th=[ 4704],  5.00th=[ 7328], 10.00th=[ 7520], 20.00th=[ 7584],
     | 30.00th=[ 7648], 40.00th=[ 7712], 50.00th=[ 7776], 60.00th=[ 7840],
     | 70.00th=[ 7904], 80.00th=[ 7968], 90.00th=[ 8032], 95.00th=[ 8160],
     | 99.00th=[15552], 99.50th=[19584], 99.90th=[25216], 99.95th=[25472],
     | 99.99th=[68096]
   bw (  KiB/s): min=101229, max=101973, per=100.00%, avg=101532.57, stdev=208.46, samples=14
   iops        : min=202458, max=203946, avg=203065.21, stdev=416.85, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 4=0.01%, 10=96.86%, 20=2.76%, 50=0.35%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=22.60%, sys=57.74%, ctx=681640, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1421257,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=99.1MiB/s (104MB/s), 99.1MiB/s-99.1MiB/s (104MB/s-104MB/s), io=694MiB (728MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751858,
  "timestamp_ms" : 1666751858044,
  "time" : "Wed Oct 26 04:37:38 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 724207104,
        "io_kbytes" : 707233,
        "bw_bytes" : 103443380,
        "bw" : 101018,
        "iops" : 202037.566062,
        "runtime" : 7001,
        "total_ios" : 1414465,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1472,
          "max" : 29145,
          "mean" : 1659.259898,
          "stddev" : 223.201329,
          "N" : 1414466
        },
        "clat_ns" : {
          "min" : 451,
          "max" : 122610,
          "mean" : 7996.446592,
          "stddev" : 1863.055498,
          "N" : 1414466,
          "percentile" : {
            "1.000000" : 4896,
            "5.000000" : 7520,
            "10.000000" : 7584,
            "20.000000" : 7584,
            "30.000000" : 7648,
            "40.000000" : 7712,
            "50.000000" : 7840,
            "60.000000" : 7840,
            "70.000000" : 7904,
            "80.000000" : 7968,
            "90.000000" : 8032,
            "95.000000" : 8256,
            "99.000000" : 15680,
            "99.500000" : 19584,
            "99.900000" : 25472,
            "99.950000" : 26240,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 5992,
          "max" : 124454,
          "mean" : 9655.706490,
          "stddev" : 1864.008513,
          "N" : 1414466
        },
        "bw_min" : 100860,
        "bw_max" : 101305,
        "bw_agg" : 100.000000,
        "bw_mean" : 101042.214286,
        "bw_dev" : 134.849305,
        "bw_samples" : 14,
        "iops_min" : 201720,
        "iops_max" : 202610,
        "iops_mean" : 202084.285714,
        "iops_stddev" : 269.778094,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 22.514286,
      "sys_cpu" : 57.471429,
      "ctx" : 693903,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010888,
        "10" : 96.823039,
        "20" : 2.759206,
        "50" : 0.380002,
        "100" : 0.020432,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=141498: Wed Oct 26 04:37:38 2022
  read: IOPS=202k, BW=98.7MiB/s (103MB/s)(691MiB/7001msec)
    slat (nsec): min=1472, max=29145, avg=1659.26, stdev=223.20
    clat (nsec): min=451, max=122610, avg=7996.45, stdev=1863.06
     lat (usec): min=5, max=124, avg= 9.66, stdev= 1.86
    clat percentiles (nsec):
     |  1.00th=[ 4896],  5.00th=[ 7520], 10.00th=[ 7584], 20.00th=[ 7584],
     | 30.00th=[ 7648], 40.00th=[ 7712], 50.00th=[ 7840], 60.00th=[ 7840],
     | 70.00th=[ 7904], 80.00th=[ 7968], 90.00th=[ 8032], 95.00th=[ 8256],
     | 99.00th=[15680], 99.50th=[19584], 99.90th=[25472], 99.95th=[26240],
     | 99.99th=[67072]
   bw (  KiB/s): min=100860, max=101305, per=100.00%, avg=101042.21, stdev=134.85, samples=14
   iops        : min=201720, max=202610, avg=202084.29, stdev=269.78, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 4=0.01%, 10=96.82%, 20=2.76%, 50=0.38%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=22.51%, sys=57.47%, ctx=693903, majf=0, minf=1
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1414465,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=98.7MiB/s (103MB/s), 98.7MiB/s-98.7MiB/s (103MB/s-103MB/s), io=691MiB (724MB), run=7001-7001msec

$ rm /tmp/fio-output_512_2_libaio-xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_2_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_2_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751871,
  "timestamp_ms" : 1666751871710,
  "time" : "Wed Oct 26 04:37:51 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 724407808,
        "io_kbytes" : 707429,
        "bw_bytes" : 103472047,
        "bw" : 101046,
        "iops" : 202093.700900,
        "runtime" : 7001,
        "total_ios" : 1414858,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1462,
          "max" : 24455,
          "mean" : 1634.391682,
          "stddev" : 230.781785,
          "N" : 1414858
        },
        "clat_ns" : {
          "min" : 451,
          "max" : 166613,
          "mean" : 8011.332859,
          "stddev" : 1891.589306,
          "N" : 1414859,
          "percentile" : {
            "1.000000" : 4960,
            "5.000000" : 7520,
            "10.000000" : 7584,
            "20.000000" : 7648,
            "30.000000" : 7712,
            "40.000000" : 7776,
            "50.000000" : 7840,
            "60.000000" : 7904,
            "70.000000" : 7904,
            "80.000000" : 7968,
            "90.000000" : 8032,
            "95.000000" : 8256,
            "99.000000" : 15808,
            "99.500000" : 19584,
            "99.900000" : 25472,
            "99.950000" : 25984,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 5981,
          "max" : 168406,
          "mean" : 9645.724596,
          "stddev" : 1891.318592,
          "N" : 1414859
        },
        "bw_min" : 100815,
        "bw_max" : 101425,
        "bw_agg" : 100.000000,
        "bw_mean" : 101073.285714,
        "bw_dev" : 170.786609,
        "bw_samples" : 14,
        "iops_min" : 201630,
        "iops_max" : 202851,
        "iops_mean" : 202146.642857,
        "iops_stddev" : 341.731700,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.114286,
      "sys_cpu" : 59.771429,
      "ctx" : 694144,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.841308,
        "20" : 2.744304,
        "50" : 0.377635,
        "100" : 0.023041,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=141538: Wed Oct 26 04:37:51 2022
  read: IOPS=202k, BW=98.7MiB/s (103MB/s)(691MiB/7001msec)
    slat (nsec): min=1462, max=24455, avg=1634.39, stdev=230.78
    clat (nsec): min=451, max=166613, avg=8011.33, stdev=1891.59
     lat (usec): min=5, max=168, avg= 9.65, stdev= 1.89
    clat percentiles (nsec):
     |  1.00th=[ 4960],  5.00th=[ 7520], 10.00th=[ 7584], 20.00th=[ 7648],
     | 30.00th=[ 7712], 40.00th=[ 7776], 50.00th=[ 7840], 60.00th=[ 7904],
     | 70.00th=[ 7904], 80.00th=[ 7968], 90.00th=[ 8032], 95.00th=[ 8256],
     | 99.00th=[15808], 99.50th=[19584], 99.90th=[25472], 99.95th=[25984],
     | 99.99th=[66048]
   bw (  KiB/s): min=100815, max=101425, per=100.00%, avg=101073.29, stdev=170.79, samples=14
   iops        : min=201630, max=202851, avg=202146.64, stdev=341.73, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 4=0.01%, 10=96.84%, 20=2.74%, 50=0.38%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=20.11%, sys=59.77%, ctx=694144, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1414858,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=98.7MiB/s (103MB/s), 98.7MiB/s-98.7MiB/s (103MB/s-103MB/s), io=691MiB (724MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751885,
  "timestamp_ms" : 1666751885332,
  "time" : "Wed Oct 26 04:38:05 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1191164928,
        "io_kbytes" : 1163247,
        "bw_bytes" : 170142112,
        "bw" : 166154,
        "iops" : 332308.241680,
        "runtime" : 7001,
        "total_ios" : 2326490,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1142,
          "max" : 25858,
          "mean" : 2197.265368,
          "stddev" : 1099.926498,
          "N" : 2326491
        },
        "clat_ns" : {
          "min" : 80,
          "max" : 165000,
          "mean" : 9370.234935,
          "stddev" : 2562.392231,
          "N" : 2326493,
          "percentile" : {
            "1.000000" : 6048,
            "5.000000" : 7648,
            "10.000000" : 8640,
            "20.000000" : 8768,
            "30.000000" : 8768,
            "40.000000" : 8768,
            "50.000000" : 8896,
            "60.000000" : 8896,
            "70.000000" : 9024,
            "80.000000" : 9152,
            "90.000000" : 10304,
            "95.000000" : 11584,
            "99.000000" : 21888,
            "99.500000" : 24192,
            "99.900000" : 28800,
            "99.950000" : 34048,
            "99.990000" : 70144
          }
        },
        "lat_ns" : {
          "min" : 5481,
          "max" : 166182,
          "mean" : 11567.500667,
          "stddev" : 2627.540346,
          "N" : 2326493
        },
        "bw_min" : 165820,
        "bw_max" : 166583,
        "bw_agg" : 100.000000,
        "bw_mean" : 166195.857143,
        "bw_dev" : 234.095923,
        "bw_samples" : 14,
        "iops_min" : 331640,
        "iops_max" : 333166,
        "iops_mean" : 332391.785714,
        "iops_stddev" : 468.114085,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.371429,
      "sys_cpu" : 82.142857,
      "ctx" : 13656,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.302774,
        "10" : 82.745681,
        "20" : 15.383217,
        "50" : 1.505831,
        "100" : 0.039759,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=141577: Wed Oct 26 04:38:05 2022
  read: IOPS=332k, BW=162MiB/s (170MB/s)(1136MiB/7001msec)
    slat (nsec): min=1142, max=25858, avg=2197.27, stdev=1099.93
    clat (nsec): min=80, max=165000, avg=9370.23, stdev=2562.39
     lat (usec): min=5, max=166, avg=11.57, stdev= 2.63
    clat percentiles (nsec):
     |  1.00th=[ 6048],  5.00th=[ 7648], 10.00th=[ 8640], 20.00th=[ 8768],
     | 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8896], 60.00th=[ 8896],
     | 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[10304], 95.00th=[11584],
     | 99.00th=[21888], 99.50th=[24192], 99.90th=[28800], 99.95th=[34048],
     | 99.99th=[70144]
   bw (  KiB/s): min=165820, max=166583, per=100.00%, avg=166195.86, stdev=234.10, samples=14
   iops        : min=331640, max=333166, avg=332391.79, stdev=468.11, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.30%, 10=82.75%, 20=15.38%, 50=1.51%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=15.37%, sys=82.14%, ctx=13656, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2326490,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=1136MiB (1191MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751898,
  "timestamp_ms" : 1666751898959,
  "time" : "Wed Oct 26 04:38:18 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1195248640,
        "io_kbytes" : 1167235,
        "bw_bytes" : 170725416,
        "bw" : 166724,
        "iops" : 333447.507499,
        "runtime" : 7001,
        "total_ios" : 2334466,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1142,
          "max" : 21070,
          "mean" : 2160.286033,
          "stddev" : 1053.646322,
          "N" : 2334467
        },
        "clat_ns" : {
          "min" : 70,
          "max" : 172754,
          "mean" : 9375.272461,
          "stddev" : 2567.897705,
          "N" : 2334469,
          "percentile" : {
            "1.000000" : 6048,
            "5.000000" : 7584,
            "10.000000" : 8640,
            "20.000000" : 8768,
            "30.000000" : 8768,
            "40.000000" : 8768,
            "50.000000" : 8896,
            "60.000000" : 8896,
            "70.000000" : 8896,
            "80.000000" : 10048,
            "90.000000" : 10304,
            "95.000000" : 11584,
            "99.000000" : 21888,
            "99.500000" : 23936,
            "99.900000" : 28288,
            "99.950000" : 33536,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 5440,
          "max" : 173937,
          "mean" : 11535.557651,
          "stddev" : 2590.223019,
          "N" : 2334469
        },
        "bw_min" : 166368,
        "bw_max" : 167296,
        "bw_agg" : 100.000000,
        "bw_mean" : 166781.142857,
        "bw_dev" : 248.585493,
        "bw_samples" : 14,
        "iops_min" : 332737,
        "iops_max" : 334592,
        "iops_mean" : 333562.357143,
        "iops_stddev" : 497.043197,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.000000,
      "sys_cpu" : 82.514286,
      "ctx" : 13822,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.012894,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.289659,
        "10" : 79.779187,
        "20" : 18.447945,
        "50" : 1.422681,
        "100" : 0.036282,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=141615: Wed Oct 26 04:38:18 2022
  read: IOPS=333k, BW=163MiB/s (171MB/s)(1140MiB/7001msec)
    slat (nsec): min=1142, max=21070, avg=2160.29, stdev=1053.65
    clat (nsec): min=70, max=172754, avg=9375.27, stdev=2567.90
     lat (usec): min=5, max=173, avg=11.54, stdev= 2.59
    clat percentiles (nsec):
     |  1.00th=[ 6048],  5.00th=[ 7584], 10.00th=[ 8640], 20.00th=[ 8768],
     | 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8896], 60.00th=[ 8896],
     | 70.00th=[ 8896], 80.00th=[10048], 90.00th=[10304], 95.00th=[11584],
     | 99.00th=[21888], 99.50th=[23936], 99.90th=[28288], 99.95th=[33536],
     | 99.99th=[72192]
   bw (  KiB/s): min=166368, max=167296, per=100.00%, avg=166781.14, stdev=248.59, samples=14
   iops        : min=332737, max=334592, avg=333562.36, stdev=497.04, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.29%, 10=79.78%, 20=18.45%, 50=1.42%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=15.00%, sys=82.51%, ctx=13822, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2334466,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=1140MiB (1195MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751912,
  "timestamp_ms" : 1666751912572,
  "time" : "Wed Oct 26 04:38:32 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1193305600,
        "io_kbytes" : 1165337,
        "bw_bytes" : 170447878,
        "bw" : 166453,
        "iops" : 332905.442080,
        "runtime" : 7001,
        "total_ios" : 2330671,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1142,
          "max" : 25989,
          "mean" : 2188.620048,
          "stddev" : 1059.038506,
          "N" : 2330672
        },
        "clat_ns" : {
          "min" : 70,
          "max" : 194865,
          "mean" : 9353.569892,
          "stddev" : 2502.558563,
          "N" : 2330674,
          "percentile" : {
            "1.000000" : 6048,
            "5.000000" : 7584,
            "10.000000" : 8640,
            "20.000000" : 8768,
            "30.000000" : 8768,
            "40.000000" : 8768,
            "50.000000" : 8896,
            "60.000000" : 8896,
            "70.000000" : 8896,
            "80.000000" : 9280,
            "90.000000" : 10304,
            "95.000000" : 11584,
            "99.000000" : 21888,
            "99.500000" : 23680,
            "99.900000" : 27264,
            "99.950000" : 32640,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5451,
          "max" : 196047,
          "mean" : 11542.189115,
          "stddev" : 2537.835066,
          "N" : 2330674
        },
        "bw_min" : 166200,
        "bw_max" : 167348,
        "bw_agg" : 100.000000,
        "bw_mean" : 166499.928571,
        "bw_dev" : 288.296713,
        "bw_samples" : 14,
        "iops_min" : 332400,
        "iops_max" : 334696,
        "iops_mean" : 332999.857143,
        "iops_stddev" : 576.593425,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.900000,
      "sys_cpu" : 83.671429,
      "ctx" : 13746,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.012014,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.293006,
        "10" : 81.953008,
        "20" : 16.269478,
        "50" : 1.433021,
        "100" : 0.029820,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=141653: Wed Oct 26 04:38:32 2022
  read: IOPS=333k, BW=163MiB/s (170MB/s)(1138MiB/7001msec)
    slat (nsec): min=1142, max=25989, avg=2188.62, stdev=1059.04
    clat (nsec): min=70, max=194865, avg=9353.57, stdev=2502.56
     lat (usec): min=5, max=196, avg=11.54, stdev= 2.54
    clat percentiles (nsec):
     |  1.00th=[ 6048],  5.00th=[ 7584], 10.00th=[ 8640], 20.00th=[ 8768],
     | 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8896], 60.00th=[ 8896],
     | 70.00th=[ 8896], 80.00th=[ 9280], 90.00th=[10304], 95.00th=[11584],
     | 99.00th=[21888], 99.50th=[23680], 99.90th=[27264], 99.95th=[32640],
     | 99.99th=[68096]
   bw (  KiB/s): min=166200, max=167348, per=100.00%, avg=166499.93, stdev=288.30, samples=14
   iops        : min=332400, max=334696, avg=332999.86, stdev=576.59, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.29%, 10=81.95%, 20=16.27%, 50=1.43%
  lat (usec)   : 100=0.03%, 250=0.01%
  cpu          : usr=13.90%, sys=83.67%, ctx=13746, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2330671,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=163MiB/s (170MB/s), 163MiB/s-163MiB/s (170MB/s-170MB/s), io=1138MiB (1193MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751023,
  "timestamp_ms" : 1666751023343,
  "time" : "Wed Oct 26 04:23:43 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 477709312,
        "io_kbytes" : 466513,
        "bw_bytes" : 68234439,
        "bw" : 66635,
        "iops" : 133270.389944,
        "runtime" : 7001,
        "total_ios" : 933026,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 250,
          "max" : 10320,
          "mean" : 279.075861,
          "stddev" : 23.844744,
          "N" : 933026
        },
        "clat_ns" : {
          "min" : 250,
          "max" : 119153,
          "mean" : 7007.570928,
          "stddev" : 1331.921014,
          "N" : 933026,
          "percentile" : {
            "1.000000" : 6624,
            "5.000000" : 6688,
            "10.000000" : 6752,
            "20.000000" : 6752,
            "30.000000" : 6752,
            "40.000000" : 6816,
            "50.000000" : 6816,
            "60.000000" : 6880,
            "70.000000" : 6880,
            "80.000000" : 6944,
            "90.000000" : 7008,
            "95.000000" : 7136,
            "99.000000" : 14400,
            "99.500000" : 17024,
            "99.900000" : 18048,
            "99.950000" : 18560,
            "99.990000" : 59136
          }
        },
        "lat_ns" : {
          "min" : 6392,
          "max" : 119454,
          "mean" : 7286.646789,
          "stddev" : 1332.043757,
          "N" : 933026
        },
        "bw_min" : 66542,
        "bw_max" : 66739,
        "bw_agg" : 100.000000,
        "bw_mean" : 66649.214286,
        "bw_dev" : 62.060986,
        "bw_samples" : 14,
        "iops_min" : 133084,
        "iops_max" : 133478,
        "iops_mean" : 133298.428571,
        "iops_stddev" : 124.121971,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 27,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.287508,
        "20" : 1.672622,
        "50" : 0.025401,
        "100" : 0.014148,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=138918: Wed Oct 26 04:23:43 2022
  read: IOPS=133k, BW=65.1MiB/s (68.2MB/s)(456MiB/7001msec)
    slat (nsec): min=250, max=10320, avg=279.08, stdev=23.84
    clat (nsec): min=250, max=119153, avg=7007.57, stdev=1331.92
     lat (usec): min=6, max=119, avg= 7.29, stdev= 1.33
    clat percentiles (nsec):
     |  1.00th=[ 6624],  5.00th=[ 6688], 10.00th=[ 6752], 20.00th=[ 6752],
     | 30.00th=[ 6752], 40.00th=[ 6816], 50.00th=[ 6816], 60.00th=[ 6880],
     | 70.00th=[ 6880], 80.00th=[ 6944], 90.00th=[ 7008], 95.00th=[ 7136],
     | 99.00th=[14400], 99.50th=[17024], 99.90th=[18048], 99.95th=[18560],
     | 99.99th=[59136]
   bw (  KiB/s): min=66542, max=66739, per=100.00%, avg=66649.21, stdev=62.06, samples=14
   iops        : min=133084, max=133478, avg=133298.43, stdev=124.12, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=98.29%, 20=1.67%, 50=0.03%, 100=0.01%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=27, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=933026,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=65.1MiB/s (68.2MB/s), 65.1MiB/s-65.1MiB/s (68.2MB/s-68.2MB/s), io=456MiB (478MB), run=7001-7001msec

$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751926,
  "timestamp_ms" : 1666751926230,
  "time" : "Wed Oct 26 04:38:46 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1147879936,
        "io_kbytes" : 1120976,
        "bw_bytes" : 163959425,
        "bw" : 160116,
        "iops" : 320232.681046,
        "runtime" : 7001,
        "total_ios" : 2241949,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1193,
          "max" : 33182,
          "mean" : 2542.839603,
          "stddev" : 751.011231,
          "N" : 2241950
        },
        "clat_ns" : {
          "min" : 120,
          "max" : 161323,
          "mean" : 9665.963411,
          "stddev" : 2557.083231,
          "N" : 2241952,
          "percentile" : {
            "1.000000" : 6368,
            "5.000000" : 8032,
            "10.000000" : 9024,
            "20.000000" : 9024,
            "30.000000" : 9152,
            "40.000000" : 9152,
            "50.000000" : 9280,
            "60.000000" : 9280,
            "70.000000" : 9280,
            "80.000000" : 9408,
            "90.000000" : 10688,
            "95.000000" : 10944,
            "99.000000" : 22912,
            "99.500000" : 24704,
            "99.900000" : 28800,
            "99.950000" : 33536,
            "99.990000" : 71168
          }
        },
        "lat_ns" : {
          "min" : 5630,
          "max" : 162565,
          "mean" : 12208.801880,
          "stddev" : 2451.667220,
          "N" : 2241952
        },
        "bw_min" : 159647,
        "bw_max" : 160652,
        "bw_agg" : 100.000000,
        "bw_mean" : 160159.214286,
        "bw_dev" : 275.571065,
        "bw_samples" : 14,
        "iops_min" : 319294,
        "iops_max" : 321304,
        "iops_mean" : 320318.500000,
        "iops_stddev" : 551.411999,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 18.742857,
      "sys_cpu" : 78.857143,
      "ctx" : 12881,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.304556,
        "10" : 85.096405,
        "20" : 12.938698,
        "50" : 1.610117,
        "100" : 0.036888,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=141691: Wed Oct 26 04:38:46 2022
  read: IOPS=320k, BW=156MiB/s (164MB/s)(1095MiB/7001msec)
    slat (nsec): min=1193, max=33182, avg=2542.84, stdev=751.01
    clat (nsec): min=120, max=161323, avg=9665.96, stdev=2557.08
     lat (usec): min=5, max=162, avg=12.21, stdev= 2.45
    clat percentiles (nsec):
     |  1.00th=[ 6368],  5.00th=[ 8032], 10.00th=[ 9024], 20.00th=[ 9024],
     | 30.00th=[ 9152], 40.00th=[ 9152], 50.00th=[ 9280], 60.00th=[ 9280],
     | 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[10688], 95.00th=[10944],
     | 99.00th=[22912], 99.50th=[24704], 99.90th=[28800], 99.95th=[33536],
     | 99.99th=[71168]
   bw (  KiB/s): min=159647, max=160652, per=100.00%, avg=160159.21, stdev=275.57, samples=14
   iops        : min=319294, max=321304, avg=320318.50, stdev=551.41, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.30%, 10=85.10%, 20=12.94%, 50=1.61%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=18.74%, sys=78.86%, ctx=12881, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2241949,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=1095MiB (1148MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751939,
  "timestamp_ms" : 1666751939860,
  "time" : "Wed Oct 26 04:38:59 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1146219008,
        "io_kbytes" : 1119354,
        "bw_bytes" : 163722183,
        "bw" : 159884,
        "iops" : 319769.318669,
        "runtime" : 7001,
        "total_ios" : 2238705,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1202,
          "max" : 26811,
          "mean" : 2540.398739,
          "stddev" : 740.364198,
          "N" : 2238706
        },
        "clat_ns" : {
          "min" : 110,
          "max" : 124784,
          "mean" : 9685.766814,
          "stddev" : 2569.003445,
          "N" : 2238708,
          "percentile" : {
            "1.000000" : 6432,
            "5.000000" : 8096,
            "10.000000" : 9024,
            "20.000000" : 9024,
            "30.000000" : 9152,
            "40.000000" : 9152,
            "50.000000" : 9280,
            "60.000000" : 9280,
            "70.000000" : 9280,
            "80.000000" : 9408,
            "90.000000" : 10688,
            "95.000000" : 10944,
            "99.000000" : 23168,
            "99.500000" : 24704,
            "99.900000" : 29568,
            "99.950000" : 34048,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 5580,
          "max" : 126016,
          "mean" : 12226.165602,
          "stddev" : 2465.670806,
          "N" : 2238708
        },
        "bw_min" : 159323,
        "bw_max" : 160812,
        "bw_agg" : 100.000000,
        "bw_mean" : 159930.214286,
        "bw_dev" : 504.450833,
        "bw_samples" : 14,
        "iops_min" : 318646,
        "iops_max" : 321625,
        "iops_mean" : 319860.500000,
        "iops_stddev" : 1009.131595,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.500000,
      "sys_cpu" : 82.100000,
      "ctx" : 12883,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.293429,
        "10" : 84.859327,
        "20" : 13.183113,
        "50" : 1.610619,
        "100" : 0.040336,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=141729: Wed Oct 26 04:38:59 2022
  read: IOPS=320k, BW=156MiB/s (164MB/s)(1093MiB/7001msec)
    slat (nsec): min=1202, max=26811, avg=2540.40, stdev=740.36
    clat (nsec): min=110, max=124784, avg=9685.77, stdev=2569.00
     lat (usec): min=5, max=126, avg=12.23, stdev= 2.47
    clat percentiles (nsec):
     |  1.00th=[ 6432],  5.00th=[ 8096], 10.00th=[ 9024], 20.00th=[ 9024],
     | 30.00th=[ 9152], 40.00th=[ 9152], 50.00th=[ 9280], 60.00th=[ 9280],
     | 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[10688], 95.00th=[10944],
     | 99.00th=[23168], 99.50th=[24704], 99.90th=[29568], 99.95th=[34048],
     | 99.99th=[72192]
   bw (  KiB/s): min=159323, max=160812, per=100.00%, avg=159930.21, stdev=504.45, samples=14
   iops        : min=318646, max=321625, avg=319860.50, stdev=1009.13, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.29%, 10=84.86%, 20=13.18%, 50=1.61%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=15.50%, sys=82.10%, ctx=12883, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2238705,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=1093MiB (1146MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751953,
  "timestamp_ms" : 1666751953498,
  "time" : "Wed Oct 26 04:39:13 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1122532864,
        "io_kbytes" : 1096223,
        "bw_bytes" : 160338932,
        "bw" : 156580,
        "iops" : 313161.405513,
        "runtime" : 7001,
        "total_ios" : 2192443,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1212,
          "max" : 24466,
          "mean" : 2454.161673,
          "stddev" : 961.853907,
          "N" : 2192444
        },
        "clat_ns" : {
          "min" : 130,
          "max" : 143108,
          "mean" : 9983.846668,
          "stddev" : 2529.342162,
          "N" : 2192446,
          "percentile" : {
            "1.000000" : 7648,
            "5.000000" : 8160,
            "10.000000" : 9280,
            "20.000000" : 9280,
            "30.000000" : 9408,
            "40.000000" : 9408,
            "50.000000" : 9536,
            "60.000000" : 9536,
            "70.000000" : 9536,
            "80.000000" : 10688,
            "90.000000" : 10944,
            "95.000000" : 11072,
            "99.000000" : 22912,
            "99.500000" : 24704,
            "99.900000" : 28544,
            "99.950000" : 33536,
            "99.990000" : 71168
          }
        },
        "lat_ns" : {
          "min" : 5500,
          "max" : 144361,
          "mean" : 12438.008588,
          "stddev" : 2576.508476,
          "N" : 2192446
        },
        "bw_min" : 156329,
        "bw_max" : 156855,
        "bw_agg" : 100.000000,
        "bw_mean" : 156627.857143,
        "bw_dev" : 180.425078,
        "bw_samples" : 14,
        "iops_min" : 312658,
        "iops_max" : 313710,
        "iops_mean" : 313255.714286,
        "iops_stddev" : 361.150177,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.500000,
      "sys_cpu" : 77.171429,
      "ctx" : 12638,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.095647,
        "10" : 78.931676,
        "20" : 19.299658,
        "50" : 1.626907,
        "100" : 0.036307,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=141767: Wed Oct 26 04:39:13 2022
  read: IOPS=313k, BW=153MiB/s (160MB/s)(1071MiB/7001msec)
    slat (nsec): min=1212, max=24466, avg=2454.16, stdev=961.85
    clat (nsec): min=130, max=143108, avg=9983.85, stdev=2529.34
     lat (usec): min=5, max=144, avg=12.44, stdev= 2.58
    clat percentiles (nsec):
     |  1.00th=[ 7648],  5.00th=[ 8160], 10.00th=[ 9280], 20.00th=[ 9280],
     | 30.00th=[ 9408], 40.00th=[ 9408], 50.00th=[ 9536], 60.00th=[ 9536],
     | 70.00th=[ 9536], 80.00th=[10688], 90.00th=[10944], 95.00th=[11072],
     | 99.00th=[22912], 99.50th=[24704], 99.90th=[28544], 99.95th=[33536],
     | 99.99th=[71168]
   bw (  KiB/s): min=156329, max=156855, per=100.00%, avg=156627.86, stdev=180.43, samples=14
   iops        : min=312658, max=313710, avg=313255.71, stdev=361.15, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.10%, 10=78.93%, 20=19.30%, 50=1.63%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=20.50%, sys=77.17%, ctx=12638, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2192443,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=1071MiB (1123MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751968,
  "timestamp_ms" : 1666751968527,
  "time" : "Wed Oct 26 04:39:28 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1505933312,
        "io_kbytes" : 1470638,
        "bw_bytes" : 215102601,
        "bw" : 210061,
        "iops" : 420121.839737,
        "runtime" : 7001,
        "total_ios" : 2941273,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 210,
          "max" : 28404,
          "mean" : 272.227385,
          "stddev" : 44.424831,
          "N" : 2941273
        },
        "clat_ns" : {
          "min" : 360,
          "max" : 125015,
          "mean" : 9017.813720,
          "stddev" : 2595.881437,
          "N" : 2941276,
          "percentile" : {
            "1.000000" : 6752,
            "5.000000" : 7456,
            "10.000000" : 7840,
            "20.000000" : 7968,
            "30.000000" : 8032,
            "40.000000" : 8160,
            "50.000000" : 8256,
            "60.000000" : 9152,
            "70.000000" : 9280,
            "80.000000" : 9408,
            "90.000000" : 9664,
            "95.000000" : 10560,
            "99.000000" : 20352,
            "99.500000" : 22144,
            "99.900000" : 29568,
            "99.950000" : 36096,
            "99.990000" : 69120
          }
        },
        "lat_ns" : {
          "min" : 5751,
          "max" : 125285,
          "mean" : 9290.041100,
          "stddev" : 2596.346238,
          "N" : 2941276
        },
        "bw_min" : 209329,
        "bw_max" : 211028,
        "bw_agg" : 100.000000,
        "bw_mean" : 210143.428571,
        "bw_dev" : 467.370338,
        "bw_samples" : 14,
        "iops_min" : 418658,
        "iops_max" : 422056,
        "iops_mean" : 420286.428571,
        "iops_stddev" : 934.806662,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 21,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 1.919441,
        "4" : 98.072977,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 93.588456,
        "20" : 5.184796,
        "50" : 1.186323,
        "100" : 0.035801,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141818: Wed Oct 26 04:39:28 2022
  read: IOPS=420k, BW=205MiB/s (215MB/s)(1436MiB/7001msec)
    slat (nsec): min=210, max=28404, avg=272.23, stdev=44.42
    clat (nsec): min=360, max=125015, avg=9017.81, stdev=2595.88
     lat (usec): min=5, max=125, avg= 9.29, stdev= 2.60
    clat percentiles (nsec):
     |  1.00th=[ 6752],  5.00th=[ 7456], 10.00th=[ 7840], 20.00th=[ 7968],
     | 30.00th=[ 8032], 40.00th=[ 8160], 50.00th=[ 8256], 60.00th=[ 9152],
     | 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9664], 95.00th=[10560],
     | 99.00th=[20352], 99.50th=[22144], 99.90th=[29568], 99.95th=[36096],
     | 99.99th=[69120]
   bw (  KiB/s): min=209329, max=211028, per=100.00%, avg=210143.43, stdev=467.37, samples=14
   iops        : min=418658, max=422056, avg=420286.43, stdev=934.81, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=93.59%, 20=5.18%, 50=1.19%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=21, majf=0, minf=0
  IO depths    : 1=0.1%, 2=1.9%, 4=98.1%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2941273,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=1436MiB (1506MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory

$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751983,
  "timestamp_ms" : 1666751983603,
  "time" : "Wed Oct 26 04:39:43 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1417665024,
        "io_kbytes" : 1384438,
        "bw_bytes" : 202494647,
        "bw" : 197748,
        "iops" : 395496.929010,
        "runtime" : 7001,
        "total_ios" : 2768874,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 170,
          "max" : 8436,
          "mean" : 264.949002,
          "stddev" : 22.607211,
          "N" : 2768874
        },
        "clat_ns" : {
          "min" : 2355,
          "max" : 122710,
          "mean" : 9554.498064,
          "stddev" : 2412.332312,
          "N" : 2768877,
          "percentile" : {
            "1.000000" : 6432,
            "5.000000" : 7392,
            "10.000000" : 8032,
            "20.000000" : 8768,
            "30.000000" : 9152,
            "40.000000" : 9152,
            "50.000000" : 9280,
            "60.000000" : 9408,
            "70.000000" : 9536,
            "80.000000" : 9792,
            "90.000000" : 10432,
            "95.000000" : 11840,
            "99.000000" : 20096,
            "99.500000" : 22144,
            "99.900000" : 28032,
            "99.950000" : 35072,
            "99.990000" : 69120
          }
        },
        "lat_ns" : {
          "min" : 5390,
          "max" : 122971,
          "mean" : 9819.447028,
          "stddev" : 2413.593380,
          "N" : 2768877
        },
        "bw_min" : 197589,
        "bw_max" : 198348,
        "bw_agg" : 100.000000,
        "bw_mean" : 197841.000000,
        "bw_dev" : 217.461933,
        "bw_samples" : 14,
        "iops_min" : 395178,
        "iops_max" : 396696,
        "iops_mean" : 395682.071429,
        "iops_stddev" : 435.142499,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 24,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 11.586082,
        "4" : 88.411318,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 87.145894,
        "20" : 11.771283,
        "50" : 1.041795,
        "100" : 0.036007,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141869: Wed Oct 26 04:39:43 2022
  read: IOPS=395k, BW=193MiB/s (202MB/s)(1352MiB/7001msec)
    slat (nsec): min=170, max=8436, avg=264.95, stdev=22.61
    clat (usec): min=2, max=122, avg= 9.55, stdev= 2.41
     lat (usec): min=5, max=122, avg= 9.82, stdev= 2.41
    clat percentiles (nsec):
     |  1.00th=[ 6432],  5.00th=[ 7392], 10.00th=[ 8032], 20.00th=[ 8768],
     | 30.00th=[ 9152], 40.00th=[ 9152], 50.00th=[ 9280], 60.00th=[ 9408],
     | 70.00th=[ 9536], 80.00th=[ 9792], 90.00th=[10432], 95.00th=[11840],
     | 99.00th=[20096], 99.50th=[22144], 99.90th=[28032], 99.95th=[35072],
     | 99.99th=[69120]
   bw (  KiB/s): min=197589, max=198348, per=100.00%, avg=197841.00, stdev=217.46, samples=14
   iops        : min=395178, max=396696, avg=395682.07, stdev=435.14, samples=14
  lat (usec)   : 4=0.01%, 10=87.15%, 20=11.77%, 50=1.04%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=24, majf=0, minf=0
  IO depths    : 1=0.1%, 2=11.6%, 4=88.4%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2768874,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=193MiB/s (202MB/s), 193MiB/s-193MiB/s (202MB/s-202MB/s), io=1352MiB (1418MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751998,
  "timestamp_ms" : 1666751998687,
  "time" : "Wed Oct 26 04:39:58 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1425394688,
        "io_kbytes" : 1391987,
        "bw_bytes" : 203598727,
        "bw" : 198826,
        "iops" : 397653.335238,
        "runtime" : 7001,
        "total_ios" : 2783971,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 180,
          "max" : 14417,
          "mean" : 263.192554,
          "stddev" : 26.016764,
          "N" : 2783971
        },
        "clat_ns" : {
          "min" : 391,
          "max" : 156223,
          "mean" : 9514.911947,
          "stddev" : 2418.859053,
          "N" : 2783974,
          "percentile" : {
            "1.000000" : 6432,
            "5.000000" : 7456,
            "10.000000" : 8160,
            "20.000000" : 8768,
            "30.000000" : 9024,
            "40.000000" : 9152,
            "50.000000" : 9280,
            "60.000000" : 9280,
            "70.000000" : 9408,
            "80.000000" : 9536,
            "90.000000" : 10176,
            "95.000000" : 11584,
            "99.000000" : 20352,
            "99.500000" : 22144,
            "99.900000" : 27776,
            "99.950000" : 34560,
            "99.990000" : 69120
          }
        },
        "lat_ns" : {
          "min" : 5100,
          "max" : 156484,
          "mean" : 9778.104498,
          "stddev" : 2419.649924,
          "N" : 2783974
        },
        "bw_min" : 198533,
        "bw_max" : 199591,
        "bw_agg" : 100.000000,
        "bw_mean" : 198952.285714,
        "bw_dev" : 317.459247,
        "bw_samples" : 14,
        "iops_min" : 397066,
        "iops_max" : 399182,
        "iops_mean" : 397904.714286,
        "iops_stddev" : 634.979519,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 16,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 7.956728,
        "4" : 92.041656,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 88.694997,
        "20" : 10.182757,
        "50" : 1.083416,
        "100" : 0.033872,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=141919: Wed Oct 26 04:39:58 2022
  read: IOPS=398k, BW=194MiB/s (204MB/s)(1359MiB/7001msec)
    slat (nsec): min=180, max=14417, avg=263.19, stdev=26.02
    clat (nsec): min=391, max=156223, avg=9514.91, stdev=2418.86
     lat (usec): min=5, max=156, avg= 9.78, stdev= 2.42
    clat percentiles (nsec):
     |  1.00th=[ 6432],  5.00th=[ 7456], 10.00th=[ 8160], 20.00th=[ 8768],
     | 30.00th=[ 9024], 40.00th=[ 9152], 50.00th=[ 9280], 60.00th=[ 9280],
     | 70.00th=[ 9408], 80.00th=[ 9536], 90.00th=[10176], 95.00th=[11584],
     | 99.00th=[20352], 99.50th=[22144], 99.90th=[27776], 99.95th=[34560],
     | 99.99th=[69120]
   bw (  KiB/s): min=198533, max=199591, per=100.00%, avg=198952.29, stdev=317.46, samples=14
   iops        : min=397066, max=399182, avg=397904.71, stdev=634.98, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=88.69%, 20=10.18%, 50=1.08%, 100=0.03%
  lat (usec)   : 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=16, majf=0, minf=0
  IO depths    : 1=0.1%, 2=8.0%, 4=92.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2783971,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=194MiB/s (204MB/s), 194MiB/s-194MiB/s (204MB/s-204MB/s), io=1359MiB (1425MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752012,
  "timestamp_ms" : 1666752012376,
  "time" : "Wed Oct 26 04:40:12 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1110281216,
        "io_kbytes" : 1084259,
        "bw_bytes" : 158588946,
        "bw" : 154872,
        "iops" : 309743.465219,
        "runtime" : 7001,
        "total_ios" : 2168514,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1292,
          "max" : 24085,
          "mean" : 2504.110085,
          "stddev" : 952.888239,
          "N" : 2168515
        },
        "clat_ns" : {
          "min" : 80,
          "max" : 198122,
          "mean" : 10032.524192,
          "stddev" : 2637.425346,
          "N" : 2168517,
          "percentile" : {
            "1.000000" : 7520,
            "5.000000" : 8032,
            "10.000000" : 9280,
            "20.000000" : 9408,
            "30.000000" : 9408,
            "40.000000" : 9408,
            "50.000000" : 9536,
            "60.000000" : 9536,
            "70.000000" : 9664,
            "80.000000" : 10688,
            "90.000000" : 11072,
            "95.000000" : 11328,
            "99.000000" : 23168,
            "99.500000" : 24960,
            "99.900000" : 29312,
            "99.950000" : 34560,
            "99.990000" : 73216
          }
        },
        "lat_ns" : {
          "min" : 5731,
          "max" : 199454,
          "mean" : 12536.634481,
          "stddev" : 2666.909686,
          "N" : 2168517
        },
        "bw_min" : 154763,
        "bw_max" : 155474,
        "bw_agg" : 100.000000,
        "bw_mean" : 154943.461538,
        "bw_dev" : 227.947953,
        "bw_samples" : 13,
        "iops_min" : 309526,
        "iops_max" : 310949,
        "iops_mean" : 309887.000000,
        "iops_stddev" : 456.089903,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.742857,
      "sys_cpu" : 82.900000,
      "ctx" : 12590,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.086188,
        "10" : 78.649711,
        "20" : 19.549194,
        "50" : 1.657863,
        "100" : 0.040166,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 3043416,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 15511,
      "write_ticks" : 0,
      "in_queue" : 15511,
      "util" : 99.035239
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=141958: Wed Oct 26 04:40:12 2022
  read: IOPS=310k, BW=151MiB/s (159MB/s)(1059MiB/7001msec)
    slat (nsec): min=1292, max=24085, avg=2504.11, stdev=952.89
    clat (nsec): min=80, max=198122, avg=10032.52, stdev=2637.43
     lat (usec): min=5, max=199, avg=12.54, stdev= 2.67
    clat percentiles (nsec):
     |  1.00th=[ 7520],  5.00th=[ 8032], 10.00th=[ 9280], 20.00th=[ 9408],
     | 30.00th=[ 9408], 40.00th=[ 9408], 50.00th=[ 9536], 60.00th=[ 9536],
     | 70.00th=[ 9664], 80.00th=[10688], 90.00th=[11072], 95.00th=[11328],
     | 99.00th=[23168], 99.50th=[24960], 99.90th=[29312], 99.95th=[34560],
     | 99.99th=[73216]
   bw (  KiB/s): min=154763, max=155474, per=100.00%, avg=154943.46, stdev=227.95, samples=13
   iops        : min=309526, max=310949, avg=309887.00, stdev=456.09, samples=13
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.09%, 10=78.65%, 20=19.55%, 50=1.66%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=14.74%, sys=82.90%, ctx=12590, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2168514,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=151MiB/s (159MB/s), 151MiB/s-151MiB/s (159MB/s-159MB/s), io=1059MiB (1110MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=3043416/0, merge=0/0, ticks=15511/0, in_queue=15511, util=99.04%

$ rm /tmp/fio-output_512_4_io_uring-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752025,
  "timestamp_ms" : 1666752025990,
  "time" : "Wed Oct 26 04:40:25 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1115557888,
        "io_kbytes" : 1089412,
        "bw_bytes" : 159342649,
        "bw" : 155608,
        "iops" : 311215.540637,
        "runtime" : 7001,
        "total_ios" : 2178820,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1292,
          "max" : 28123,
          "mean" : 2499.686404,
          "stddev" : 889.896990,
          "N" : 2178821
        },
        "clat_ns" : {
          "min" : 70,
          "max" : 142838,
          "mean" : 9952.938270,
          "stddev" : 2613.804392,
          "N" : 2178823,
          "percentile" : {
            "1.000000" : 7392,
            "5.000000" : 8032,
            "10.000000" : 9152,
            "20.000000" : 9280,
            "30.000000" : 9408,
            "40.000000" : 9408,
            "50.000000" : 9536,
            "60.000000" : 9536,
            "70.000000" : 9536,
            "80.000000" : 10432,
            "90.000000" : 11072,
            "95.000000" : 11328,
            "99.000000" : 23168,
            "99.500000" : 24960,
            "99.900000" : 29824,
            "99.950000" : 34560,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 5541,
          "max" : 144180,
          "mean" : 12452.624922,
          "stddev" : 2606.970828,
          "N" : 2178823
        },
        "bw_min" : 155411,
        "bw_max" : 156036,
        "bw_agg" : 100.000000,
        "bw_mean" : 155680.285714,
        "bw_dev" : 179.364017,
        "bw_samples" : 14,
        "iops_min" : 310822,
        "iops_max" : 312072,
        "iops_mean" : 311360.571429,
        "iops_stddev" : 358.728034,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.085714,
      "sys_cpu" : 82.542857,
      "ctx" : 12650,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010281,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.110702,
        "10" : 79.515885,
        "20" : 18.645368,
        "50" : 1.666728,
        "100" : 0.042362,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 3132143,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 15922,
      "write_ticks" : 0,
      "in_queue" : 15922,
      "util" : 99.098564
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=141996: Wed Oct 26 04:40:25 2022
  read: IOPS=311k, BW=152MiB/s (159MB/s)(1064MiB/7001msec)
    slat (nsec): min=1292, max=28123, avg=2499.69, stdev=889.90
    clat (nsec): min=70, max=142838, avg=9952.94, stdev=2613.80
     lat (usec): min=5, max=144, avg=12.45, stdev= 2.61
    clat percentiles (nsec):
     |  1.00th=[ 7392],  5.00th=[ 8032], 10.00th=[ 9152], 20.00th=[ 9280],
     | 30.00th=[ 9408], 40.00th=[ 9408], 50.00th=[ 9536], 60.00th=[ 9536],
     | 70.00th=[ 9536], 80.00th=[10432], 90.00th=[11072], 95.00th=[11328],
     | 99.00th=[23168], 99.50th=[24960], 99.90th=[29824], 99.95th=[34560],
     | 99.99th=[72192]
   bw (  KiB/s): min=155411, max=156036, per=100.00%, avg=155680.29, stdev=179.36, samples=14
   iops        : min=310822, max=312072, avg=311360.57, stdev=358.73, samples=14
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.11%, 10=79.52%, 20=18.65%, 50=1.67%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=15.09%, sys=82.54%, ctx=12650, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2178820,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=1064MiB (1116MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=3132143/0, merge=0/0, ticks=15922/0, in_queue=15922, util=99.10%

$ rm /tmp/fio-output_512_4_io_uring-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752039,
  "timestamp_ms" : 1666752039618,
  "time" : "Wed Oct 26 04:40:39 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1114947584,
        "io_kbytes" : 1088816,
        "bw_bytes" : 159255475,
        "bw" : 155522,
        "iops" : 311045.279246,
        "runtime" : 7001,
        "total_ios" : 2177628,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1282,
          "max" : 115326,
          "mean" : 2502.982368,
          "stddev" : 919.655007,
          "N" : 2177629
        },
        "clat_ns" : {
          "min" : 70,
          "max" : 153448,
          "mean" : 9965.891609,
          "stddev" : 2636.066476,
          "N" : 2177631,
          "percentile" : {
            "1.000000" : 7456,
            "5.000000" : 8032,
            "10.000000" : 9152,
            "20.000000" : 9280,
            "30.000000" : 9408,
            "40.000000" : 9408,
            "50.000000" : 9536,
            "60.000000" : 9536,
            "70.000000" : 9536,
            "80.000000" : 9920,
            "90.000000" : 11072,
            "95.000000" : 11200,
            "99.000000" : 23168,
            "99.500000" : 24960,
            "99.900000" : 29312,
            "99.950000" : 34048,
            "99.990000" : 73216
          }
        },
        "lat_ns" : {
          "min" : 5721,
          "max" : 154770,
          "mean" : 12468.873560,
          "stddev" : 2639.118189,
          "N" : 2177631
        },
        "bw_min" : 155227,
        "bw_max" : 156354,
        "bw_agg" : 100.000000,
        "bw_mean" : 155585.285714,
        "bw_dev" : 335.643453,
        "bw_samples" : 14,
        "iops_min" : 310454,
        "iops_max" : 312710,
        "iops_mean" : 311170.571429,
        "iops_stddev" : 671.748776,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.314286,
      "sys_cpu" : 82.328571,
      "ctx" : 12629,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.015154,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.107319,
        "10" : 79.928757,
        "20" : 18.224600,
        "50" : 1.676457,
        "100" : 0.037380,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 3131615,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 15931,
      "write_ticks" : 0,
      "in_queue" : 15931,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=142034: Wed Oct 26 04:40:39 2022
  read: IOPS=311k, BW=152MiB/s (159MB/s)(1063MiB/7001msec)
    slat (nsec): min=1282, max=115326, avg=2502.98, stdev=919.66
    clat (nsec): min=70, max=153448, avg=9965.89, stdev=2636.07
     lat (usec): min=5, max=154, avg=12.47, stdev= 2.64
    clat percentiles (nsec):
     |  1.00th=[ 7456],  5.00th=[ 8032], 10.00th=[ 9152], 20.00th=[ 9280],
     | 30.00th=[ 9408], 40.00th=[ 9408], 50.00th=[ 9536], 60.00th=[ 9536],
     | 70.00th=[ 9536], 80.00th=[ 9920], 90.00th=[11072], 95.00th=[11200],
     | 99.00th=[23168], 99.50th=[24960], 99.90th=[29312], 99.95th=[34048],
     | 99.99th=[73216]
   bw (  KiB/s): min=155227, max=156354, per=100.00%, avg=155585.29, stdev=335.64, samples=14
   iops        : min=310454, max=312710, avg=311170.57, stdev=671.75, samples=14
  lat (nsec)   : 100=0.02%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.11%, 10=79.93%, 20=18.22%, 50=1.68%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=15.31%, sys=82.33%, ctx=12629, majf=0, minf=1
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2177628,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=1063MiB (1115MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=3131615/0, merge=0/0, ticks=15931/0, in_queue=15931, util=99.09%

$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752054,
  "timestamp_ms" : 1666752054612,
  "time" : "Wed Oct 26 04:40:54 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1012919808,
        "io_kbytes" : 989179,
        "bw_bytes" : 144682160,
        "bw" : 141291,
        "iops" : 282581.916869,
        "runtime" : 7001,
        "total_ios" : 1978356,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 26279,
          "mean" : 141.316532,
          "stddev" : 49.171458,
          "N" : 1978356
        },
        "clat_ns" : {
          "min" : 11882,
          "max" : 177744,
          "mean" : 13172.391741,
          "stddev" : 2083.866987,
          "N" : 1978356,
          "percentile" : {
            "1.000000" : 12224,
            "5.000000" : 12352,
            "10.000000" : 12352,
            "20.000000" : 12608,
            "30.000000" : 12608,
            "40.000000" : 12736,
            "50.000000" : 12864,
            "60.000000" : 12992,
            "70.000000" : 13120,
            "80.000000" : 13248,
            "90.000000" : 13376,
            "95.000000" : 14016,
            "99.000000" : 22912,
            "99.500000" : 23936,
            "99.900000" : 30592,
            "99.950000" : 46336,
            "99.990000" : 71168
          }
        },
        "lat_ns" : {
          "min" : 12003,
          "max" : 177944,
          "mean" : 13313.708273,
          "stddev" : 2088.192140,
          "N" : 1978356
        },
        "bw_min" : 141168,
        "bw_max" : 141492,
        "bw_agg" : 100.000000,
        "bw_mean" : 141330.928571,
        "bw_dev" : 101.098934,
        "bw_samples" : 14,
        "iops_min" : 282336,
        "iops_max" : 282984,
        "iops_mean" : 282661.928571,
        "iops_stddev" : 201.869137,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 38.900000,
      "sys_cpu" : 61.071429,
      "ctx" : 22,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 25.000000,
        "2" : 50.000000,
        "4" : 25.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 98.583622,
        "50" : 1.369925,
        "100" : 0.039477,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=142084: Wed Oct 26 04:40:54 2022
  read: IOPS=283k, BW=138MiB/s (145MB/s)(966MiB/7001msec)
    slat (nsec): min=110, max=26279, avg=141.32, stdev=49.17
    clat (usec): min=11, max=177, avg=13.17, stdev= 2.08
     lat (usec): min=12, max=177, avg=13.31, stdev= 2.09
    clat percentiles (nsec):
     |  1.00th=[12224],  5.00th=[12352], 10.00th=[12352], 20.00th=[12608],
     | 30.00th=[12608], 40.00th=[12736], 50.00th=[12864], 60.00th=[12992],
     | 70.00th=[13120], 80.00th=[13248], 90.00th=[13376], 95.00th=[14016],
     | 99.00th=[22912], 99.50th=[23936], 99.90th=[30592], 99.95th=[46336],
     | 99.99th=[71168]
   bw (  KiB/s): min=141168, max=141492, per=100.00%, avg=141330.93, stdev=101.10, samples=14
   iops        : min=282336, max=282984, avg=282661.93, stdev=201.87, samples=14
  lat (usec)   : 20=98.58%, 50=1.37%, 100=0.04%, 250=0.01%
  cpu          : usr=38.90%, sys=61.07%, ctx=22, majf=0, minf=0
  IO depths    : 1=25.0%, 2=50.0%, 4=25.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1978356,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=966MiB (1013MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751038,
  "timestamp_ms" : 1666751038438,
  "time" : "Wed Oct 26 04:23:58 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 438721024,
        "io_kbytes" : 428438,
        "bw_bytes" : 62665479,
        "bw" : 61196,
        "iops" : 122393.372375,
        "runtime" : 7001,
        "total_ios" : 856876,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 711,
          "max" : 19336,
          "mean" : 922.608556,
          "stddev" : 65.998960,
          "N" : 856877
        },
        "clat_ns" : {
          "min" : 731,
          "max" : 1292413,
          "mean" : 6966.034914,
          "stddev" : 1782.190180,
          "N" : 856876,
          "percentile" : {
            "1.000000" : 6368,
            "5.000000" : 6496,
            "10.000000" : 6560,
            "20.000000" : 6624,
            "30.000000" : 6752,
            "40.000000" : 6752,
            "50.000000" : 6816,
            "60.000000" : 6944,
            "70.000000" : 6944,
            "80.000000" : 7008,
            "90.000000" : 7136,
            "95.000000" : 7264,
            "99.000000" : 12864,
            "99.500000" : 13888,
            "99.900000" : 16512,
            "99.950000" : 16768,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 6993,
          "max" : 1293966,
          "mean" : 7888.643483,
          "stddev" : 1784.824087,
          "N" : 856876
        },
        "bw_min" : 61040,
        "bw_max" : 61478,
        "bw_agg" : 100.000000,
        "bw_mean" : 61222.769231,
        "bw_dev" : 152.265751,
        "bw_samples" : 13,
        "iops_min" : 122080,
        "iops_max" : 122956,
        "iops_mean" : 122445.769231,
        "iops_stddev" : 304.518241,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 21,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.313175,
        "20" : 1.668036,
        "50" : 0.010000,
        "100" : 0.013771,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=138968: Wed Oct 26 04:23:58 2022
  read: IOPS=122k, BW=59.8MiB/s (62.7MB/s)(418MiB/7001msec)
    slat (nsec): min=711, max=19336, avg=922.61, stdev=66.00
    clat (nsec): min=731, max=1292.4k, avg=6966.03, stdev=1782.19
     lat (usec): min=6, max=1293, avg= 7.89, stdev= 1.78
    clat percentiles (nsec):
     |  1.00th=[ 6368],  5.00th=[ 6496], 10.00th=[ 6560], 20.00th=[ 6624],
     | 30.00th=[ 6752], 40.00th=[ 6752], 50.00th=[ 6816], 60.00th=[ 6944],
     | 70.00th=[ 6944], 80.00th=[ 7008], 90.00th=[ 7136], 95.00th=[ 7264],
     | 99.00th=[12864], 99.50th=[13888], 99.90th=[16512], 99.95th=[16768],
     | 99.99th=[58112]
   bw (  KiB/s): min=61040, max=61478, per=100.00%, avg=61222.77, stdev=152.27, samples=13
   iops        : min=122080, max=122956, avg=122445.77, stdev=304.52, samples=13
  lat (nsec)   : 750=0.01%
  lat (usec)   : 10=98.31%, 20=1.67%, 50=0.01%, 100=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=21, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=856876,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=59.8MiB/s (62.7MB/s), 59.8MiB/s-59.8MiB/s (62.7MB/s-62.7MB/s), io=418MiB (439MB), run=7001-7001msec

$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752069,
  "timestamp_ms" : 1666752069718,
  "time" : "Wed Oct 26 04:41:09 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1013155328,
        "io_kbytes" : 989409,
        "bw_bytes" : 144715801,
        "bw" : 141324,
        "iops" : 282647.621768,
        "runtime" : 7001,
        "total_ios" : 1978816,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 12503,
          "mean" : 142.496506,
          "stddev" : 47.180392,
          "N" : 1978816
        },
        "clat_ns" : {
          "min" : 11873,
          "max" : 117370,
          "mean" : 13154.827341,
          "stddev" : 1841.054914,
          "N" : 1978816,
          "percentile" : {
            "1.000000" : 12224,
            "5.000000" : 12352,
            "10.000000" : 12352,
            "20.000000" : 12608,
            "30.000000" : 12608,
            "40.000000" : 12736,
            "50.000000" : 12864,
            "60.000000" : 12992,
            "70.000000" : 13120,
            "80.000000" : 13248,
            "90.000000" : 13376,
            "95.000000" : 13888,
            "99.000000" : 22912,
            "99.500000" : 23936,
            "99.900000" : 29568,
            "99.950000" : 35072,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 11993,
          "max" : 117570,
          "mean" : 13297.323847,
          "stddev" : 1846.116285,
          "N" : 1978816
        },
        "bw_min" : 141129,
        "bw_max" : 141560,
        "bw_agg" : 100.000000,
        "bw_mean" : 141360.928571,
        "bw_dev" : 153.096483,
        "bw_samples" : 14,
        "iops_min" : 282256,
        "iops_max" : 283120,
        "iops_mean" : 282722.000000,
        "iops_stddev" : 306.117122,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 41.242857,
      "sys_cpu" : 58.728571,
      "ctx" : 20,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 25.000000,
        "2" : 50.000000,
        "4" : 25.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 98.730200,
        "50" : 1.228361,
        "100" : 0.041237,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=142133: Wed Oct 26 04:41:09 2022
  read: IOPS=283k, BW=138MiB/s (145MB/s)(966MiB/7001msec)
    slat (nsec): min=110, max=12503, avg=142.50, stdev=47.18
    clat (usec): min=11, max=117, avg=13.15, stdev= 1.84
     lat (usec): min=11, max=117, avg=13.30, stdev= 1.85
    clat percentiles (nsec):
     |  1.00th=[12224],  5.00th=[12352], 10.00th=[12352], 20.00th=[12608],
     | 30.00th=[12608], 40.00th=[12736], 50.00th=[12864], 60.00th=[12992],
     | 70.00th=[13120], 80.00th=[13248], 90.00th=[13376], 95.00th=[13888],
     | 99.00th=[22912], 99.50th=[23936], 99.90th=[29568], 99.95th=[35072],
     | 99.99th=[66048]
   bw (  KiB/s): min=141129, max=141560, per=100.00%, avg=141360.93, stdev=153.10, samples=14
   iops        : min=282256, max=283120, avg=282722.00, stdev=306.12, samples=14
  lat (usec)   : 20=98.73%, 50=1.23%, 100=0.04%, 250=0.01%
  cpu          : usr=41.24%, sys=58.73%, ctx=20, majf=0, minf=0
  IO depths    : 1=25.0%, 2=50.0%, 4=25.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1978816,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=966MiB (1013MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-bdev_uring_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752084,
  "timestamp_ms" : 1666752084789,
  "time" : "Wed Oct 26 04:41:24 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1017103872,
        "io_kbytes" : 993265,
        "bw_bytes" : 145279798,
        "bw" : 141874,
        "iops" : 283749.178689,
        "runtime" : 7001,
        "total_ios" : 1986528,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 13124,
          "mean" : 141.327949,
          "stddev" : 46.350036,
          "N" : 1986528
        },
        "clat_ns" : {
          "min" : 11853,
          "max" : 118873,
          "mean" : 13131.754659,
          "stddev" : 1847.869899,
          "N" : 1986528,
          "percentile" : {
            "1.000000" : 12224,
            "5.000000" : 12352,
            "10.000000" : 12352,
            "20.000000" : 12480,
            "30.000000" : 12608,
            "40.000000" : 12736,
            "50.000000" : 12864,
            "60.000000" : 12992,
            "70.000000" : 12992,
            "80.000000" : 13120,
            "90.000000" : 13376,
            "95.000000" : 13888,
            "99.000000" : 22912,
            "99.500000" : 23936,
            "99.900000" : 29568,
            "99.950000" : 35072,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 11963,
          "max" : 119083,
          "mean" : 13273.082608,
          "stddev" : 1853.093830,
          "N" : 1986528
        },
        "bw_min" : 141696,
        "bw_max" : 142276,
        "bw_agg" : 100.000000,
        "bw_mean" : 141915.000000,
        "bw_dev" : 201.676817,
        "bw_samples" : 14,
        "iops_min" : 283392,
        "iops_max" : 284552,
        "iops_mean" : 283829.928571,
        "iops_stddev" : 403.275906,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 38.557143,
      "sys_cpu" : 61.414286,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 25.000000,
        "2" : 50.000000,
        "4" : 25.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 98.732412,
        "50" : 1.227670,
        "100" : 0.039718,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=142182: Wed Oct 26 04:41:24 2022
  read: IOPS=284k, BW=139MiB/s (145MB/s)(970MiB/7001msec)
    slat (nsec): min=110, max=13124, avg=141.33, stdev=46.35
    clat (usec): min=11, max=118, avg=13.13, stdev= 1.85
     lat (usec): min=11, max=119, avg=13.27, stdev= 1.85
    clat percentiles (nsec):
     |  1.00th=[12224],  5.00th=[12352], 10.00th=[12352], 20.00th=[12480],
     | 30.00th=[12608], 40.00th=[12736], 50.00th=[12864], 60.00th=[12992],
     | 70.00th=[12992], 80.00th=[13120], 90.00th=[13376], 95.00th=[13888],
     | 99.00th=[22912], 99.50th=[23936], 99.90th=[29568], 99.95th=[35072],
     | 99.99th=[66048]
   bw (  KiB/s): min=141696, max=142276, per=100.00%, avg=141915.00, stdev=201.68, samples=14
   iops        : min=283392, max=284552, avg=283829.93, stdev=403.28, samples=14
  lat (usec)   : 20=98.73%, 50=1.23%, 100=0.04%, 250=0.01%
  cpu          : usr=38.56%, sys=61.41%, ctx=23, majf=0, minf=0
  IO depths    : 1=25.0%, 2=50.0%, 4=25.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1986528,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=139MiB/s (145MB/s), 139MiB/s-139MiB/s (145MB/s-145MB/s), io=970MiB (1017MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752099,
  "timestamp_ms" : 1666752099916,
  "time" : "Wed Oct 26 04:41:39 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1077817344,
        "io_kbytes" : 1052556,
        "bw_bytes" : 153951913,
        "bw" : 150343,
        "iops" : 300686.759034,
        "runtime" : 7001,
        "total_ios" : 2105108,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1532,
          "max" : 20038,
          "mean" : 2804.261997,
          "stddev" : 1140.383516,
          "N" : 2105109
        },
        "clat_ns" : {
          "min" : 190,
          "max" : 124213,
          "mean" : 10260.727332,
          "stddev" : 2255.624306,
          "N" : 2105111,
          "percentile" : {
            "1.000000" : 8512,
            "5.000000" : 8640,
            "10.000000" : 8896,
            "20.000000" : 9792,
            "30.000000" : 9792,
            "40.000000" : 9920,
            "50.000000" : 9920,
            "60.000000" : 9920,
            "70.000000" : 10048,
            "80.000000" : 10176,
            "90.000000" : 11200,
            "95.000000" : 11456,
            "99.000000" : 20864,
            "99.500000" : 23424,
            "99.900000" : 29824,
            "99.950000" : 38144,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 5651,
          "max" : 125786,
          "mean" : 13064.989316,
          "stddev" : 2336.079254,
          "N" : 2105111
        },
        "bw_min" : 150030,
        "bw_max" : 150660,
        "bw_agg" : 100.000000,
        "bw_mean" : 150378.142857,
        "bw_dev" : 229.647508,
        "bw_samples" : 14,
        "iops_min" : 300060,
        "iops_max" : 301320,
        "iops_mean" : 300756.285714,
        "iops_stddev" : 459.295016,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.271429,
      "sys_cpu" : 67.700000,
      "ctx" : 21,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.013824,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.021139,
        "10" : 66.454738,
        "20" : 32.398053,
        "50" : 1.065931,
        "100" : 0.044938,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142231: Wed Oct 26 04:41:39 2022
  read: IOPS=301k, BW=147MiB/s (154MB/s)(1028MiB/7001msec)
    slat (nsec): min=1532, max=20038, avg=2804.26, stdev=1140.38
    clat (nsec): min=190, max=124213, avg=10260.73, stdev=2255.62
     lat (usec): min=5, max=125, avg=13.06, stdev= 2.34
    clat percentiles (nsec):
     |  1.00th=[ 8512],  5.00th=[ 8640], 10.00th=[ 8896], 20.00th=[ 9792],
     | 30.00th=[ 9792], 40.00th=[ 9920], 50.00th=[ 9920], 60.00th=[ 9920],
     | 70.00th=[10048], 80.00th=[10176], 90.00th=[11200], 95.00th=[11456],
     | 99.00th=[20864], 99.50th=[23424], 99.90th=[29824], 99.95th=[38144],
     | 99.99th=[68096]
   bw (  KiB/s): min=150030, max=150660, per=100.00%, avg=150378.14, stdev=229.65, samples=14
   iops        : min=300060, max=301320, avg=300756.29, stdev=459.30, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.02%, 10=66.45%, 20=32.40%, 50=1.07%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=32.27%, sys=67.70%, ctx=21, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2105108,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=1028MiB (1078MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752115,
  "timestamp_ms" : 1666752115189,
  "time" : "Wed Oct 26 04:41:55 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1060443136,
        "io_kbytes" : 1035589,
        "bw_bytes" : 151470237,
        "bw" : 147920,
        "iops" : 295839.737180,
        "runtime" : 7001,
        "total_ios" : 2071174,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1542,
          "max" : 28865,
          "mean" : 2851.573909,
          "stddev" : 1251.286691,
          "N" : 2071175
        },
        "clat_ns" : {
          "min" : 190,
          "max" : 120766,
          "mean" : 10416.253545,
          "stddev" : 2328.765258,
          "N" : 2071177,
          "percentile" : {
            "1.000000" : 8640,
            "5.000000" : 8768,
            "10.000000" : 8768,
            "20.000000" : 8896,
            "30.000000" : 9024,
            "40.000000" : 9920,
            "50.000000" : 10048,
            "60.000000" : 10176,
            "70.000000" : 11072,
            "80.000000" : 11328,
            "90.000000" : 11456,
            "95.000000" : 11584,
            "99.000000" : 20864,
            "99.500000" : 23424,
            "99.900000" : 29824,
            "99.950000" : 34048,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 5650,
          "max" : 122359,
          "mean" : 13267.826877,
          "stddev" : 2220.362076,
          "N" : 2071177
        },
        "bw_min" : 147590,
        "bw_max" : 148421,
        "bw_agg" : 100.000000,
        "bw_mean" : 147953.571429,
        "bw_dev" : 238.447451,
        "bw_samples" : 14,
        "iops_min" : 295180,
        "iops_max" : 296842,
        "iops_mean" : 295907.071429,
        "iops_stddev" : 476.984513,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 30.214286,
      "sys_cpu" : 69.757143,
      "ctx" : 22,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.016512,
        "10" : 42.578412,
        "20" : 56.246361,
        "50" : 1.119945,
        "100" : 0.037129,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142280: Wed Oct 26 04:41:55 2022
  read: IOPS=296k, BW=144MiB/s (151MB/s)(1011MiB/7001msec)
    slat (nsec): min=1542, max=28865, avg=2851.57, stdev=1251.29
    clat (nsec): min=190, max=120766, avg=10416.25, stdev=2328.77
     lat (usec): min=5, max=122, avg=13.27, stdev= 2.22
    clat percentiles (nsec):
     |  1.00th=[ 8640],  5.00th=[ 8768], 10.00th=[ 8768], 20.00th=[ 8896],
     | 30.00th=[ 9024], 40.00th=[ 9920], 50.00th=[10048], 60.00th=[10176],
     | 70.00th=[11072], 80.00th=[11328], 90.00th=[11456], 95.00th=[11584],
     | 99.00th=[20864], 99.50th=[23424], 99.90th=[29824], 99.95th=[34048],
     | 99.99th=[66048]
   bw (  KiB/s): min=147590, max=148421, per=100.00%, avg=147953.57, stdev=238.45, samples=14
   iops        : min=295180, max=296842, avg=295907.07, stdev=476.98, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.02%, 10=42.58%, 20=56.25%, 50=1.12%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=30.21%, sys=69.76%, ctx=22, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2071174,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=1011MiB (1060MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory

$ rm /tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752130,
  "timestamp_ms" : 1666752130258,
  "time" : "Wed Oct 26 04:42:10 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1063592960,
        "io_kbytes" : 1038665,
        "bw_bytes" : 151920148,
        "bw" : 148359,
        "iops" : 296718.468790,
        "runtime" : 7001,
        "total_ios" : 2077326,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1533,
          "max" : 57908,
          "mean" : 2828.639420,
          "stddev" : 1225.631029,
          "N" : 2077327
        },
        "clat_ns" : {
          "min" : 210,
          "max" : 121017,
          "mean" : 10398.118086,
          "stddev" : 2330.860370,
          "N" : 2077329,
          "percentile" : {
            "1.000000" : 8640,
            "5.000000" : 8768,
            "10.000000" : 8768,
            "20.000000" : 8896,
            "30.000000" : 9920,
            "40.000000" : 9920,
            "50.000000" : 10048,
            "60.000000" : 10176,
            "70.000000" : 10944,
            "80.000000" : 11200,
            "90.000000" : 11328,
            "95.000000" : 11584,
            "99.000000" : 20864,
            "99.500000" : 23424,
            "99.900000" : 30080,
            "99.950000" : 35072,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 5670,
          "max" : 122710,
          "mean" : 13226.758125,
          "stddev" : 2280.945948,
          "N" : 2077329
        },
        "bw_min" : 147572,
        "bw_max" : 148824,
        "bw_agg" : 100.000000,
        "bw_mean" : 148401.285714,
        "bw_dev" : 298.527522,
        "bw_samples" : 14,
        "iops_min" : 295144,
        "iops_max" : 297649,
        "iops_mean" : 296802.642857,
        "iops_stddev" : 597.164017,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.328571,
      "sys_cpu" : 67.642857,
      "ctx" : 29,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.023925,
        "10" : 44.046240,
        "20" : 54.765116,
        "50" : 1.121779,
        "100" : 0.039426,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142330: Wed Oct 26 04:42:10 2022
  read: IOPS=297k, BW=145MiB/s (152MB/s)(1014MiB/7001msec)
    slat (nsec): min=1533, max=57908, avg=2828.64, stdev=1225.63
    clat (nsec): min=210, max=121017, avg=10398.12, stdev=2330.86
     lat (usec): min=5, max=122, avg=13.23, stdev= 2.28
    clat percentiles (nsec):
     |  1.00th=[ 8640],  5.00th=[ 8768], 10.00th=[ 8768], 20.00th=[ 8896],
     | 30.00th=[ 9920], 40.00th=[ 9920], 50.00th=[10048], 60.00th=[10176],
     | 70.00th=[10944], 80.00th=[11200], 90.00th=[11328], 95.00th=[11584],
     | 99.00th=[20864], 99.50th=[23424], 99.90th=[30080], 99.95th=[35072],
     | 99.99th=[67072]
   bw (  KiB/s): min=147572, max=148824, per=100.00%, avg=148401.29, stdev=298.53, samples=14
   iops        : min=295144, max=297649, avg=296802.64, stdev=597.16, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.02%, 10=44.05%, 20=54.77%, 50=1.12%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=32.33%, sys=67.64%, ctx=29, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2077326,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1014MiB (1064MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752143,
  "timestamp_ms" : 1666752143949,
  "time" : "Wed Oct 26 04:42:23 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1086656512,
        "io_kbytes" : 1061188,
        "bw_bytes" : 155214471,
        "bw" : 151576,
        "iops" : 303152.692473,
        "runtime" : 7001,
        "total_ios" : 2122372,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1372,
          "max" : 112582,
          "mean" : 2810.550236,
          "stddev" : 1247.731970,
          "N" : 2122373
        },
        "clat_ns" : {
          "min" : 110,
          "max" : 125957,
          "mean" : 10142.478753,
          "stddev" : 2569.151273,
          "N" : 2122375,
          "percentile" : {
            "1.000000" : 6880,
            "5.000000" : 7968,
            "10.000000" : 8256,
            "20.000000" : 9536,
            "30.000000" : 9664,
            "40.000000" : 9664,
            "50.000000" : 9792,
            "60.000000" : 9792,
            "70.000000" : 9792,
            "80.000000" : 10048,
            "90.000000" : 11456,
            "95.000000" : 11968,
            "99.000000" : 23424,
            "99.500000" : 24960,
            "99.900000" : 29824,
            "99.950000" : 34048,
            "99.990000" : 70144
          }
        },
        "lat_ns" : {
          "min" : 5921,
          "max" : 127369,
          "mean" : 12953.029064,
          "stddev" : 2798.630057,
          "N" : 2122375
        },
        "bw_min" : 151380,
        "bw_max" : 151845,
        "bw_agg" : 100.000000,
        "bw_mean" : 151616.571429,
        "bw_dev" : 148.988236,
        "bw_samples" : 14,
        "iops_min" : 302760,
        "iops_max" : 303691,
        "iops_mean" : 303233.214286,
        "iops_stddev" : 298.105861,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 18.328571,
      "sys_cpu" : 79.457143,
      "ctx" : 12395,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.024925,
        "4" : 0.110160,
        "10" : 80.312170,
        "20" : 17.752307,
        "50" : 1.758127,
        "100" : 0.039578,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=142369: Wed Oct 26 04:42:23 2022
  read: IOPS=303k, BW=148MiB/s (155MB/s)(1036MiB/7001msec)
    slat (nsec): min=1372, max=112582, avg=2810.55, stdev=1247.73
    clat (nsec): min=110, max=125957, avg=10142.48, stdev=2569.15
     lat (usec): min=5, max=127, avg=12.95, stdev= 2.80
    clat percentiles (nsec):
     |  1.00th=[ 6880],  5.00th=[ 7968], 10.00th=[ 8256], 20.00th=[ 9536],
     | 30.00th=[ 9664], 40.00th=[ 9664], 50.00th=[ 9792], 60.00th=[ 9792],
     | 70.00th=[ 9792], 80.00th=[10048], 90.00th=[11456], 95.00th=[11968],
     | 99.00th=[23424], 99.50th=[24960], 99.90th=[29824], 99.95th=[34048],
     | 99.99th=[70144]
   bw (  KiB/s): min=151380, max=151845, per=100.00%, avg=151616.57, stdev=148.99, samples=14
   iops        : min=302760, max=303691, avg=303233.21, stdev=298.11, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.02%, 4=0.11%, 10=80.31%, 20=17.75%, 50=1.76%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=18.33%, sys=79.46%, ctx=12395, majf=0, minf=1
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2122372,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1036MiB (1087MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752157,
  "timestamp_ms" : 1666752157934,
  "time" : "Wed Oct 26 04:42:37 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1083688448,
        "io_kbytes" : 1058289,
        "bw_bytes" : 154790522,
        "bw" : 151162,
        "iops" : 302324.667905,
        "runtime" : 7001,
        "total_ios" : 2116575,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1362,
          "max" : 21871,
          "mean" : 2810.793388,
          "stddev" : 1238.113474,
          "N" : 2116576
        },
        "clat_ns" : {
          "min" : 110,
          "max" : 126127,
          "mean" : 10178.077711,
          "stddev" : 2575.128921,
          "N" : 2116578,
          "percentile" : {
            "1.000000" : 6880,
            "5.000000" : 7968,
            "10.000000" : 8512,
            "20.000000" : 9536,
            "30.000000" : 9664,
            "40.000000" : 9664,
            "50.000000" : 9792,
            "60.000000" : 9792,
            "70.000000" : 9920,
            "80.000000" : 10048,
            "90.000000" : 11456,
            "95.000000" : 11968,
            "99.000000" : 23424,
            "99.500000" : 24960,
            "99.900000" : 30336,
            "99.950000" : 34560,
            "99.990000" : 71168
          }
        },
        "lat_ns" : {
          "min" : 5841,
          "max" : 127520,
          "mean" : 12988.870477,
          "stddev" : 2788.918764,
          "N" : 2116578
        },
        "bw_min" : 150921,
        "bw_max" : 151477,
        "bw_agg" : 100.000000,
        "bw_mean" : 151202.642857,
        "bw_dev" : 172.050801,
        "bw_samples" : 14,
        "iops_min" : 301842,
        "iops_max" : 302954,
        "iops_mean" : 302405.500000,
        "iops_stddev" : 344.057408,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.585714,
      "sys_cpu" : 80.171429,
      "ctx" : 12447,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.019702,
        "4" : 0.111123,
        "10" : 79.619905,
        "20" : 18.496203,
        "50" : 1.703979,
        "100" : 0.044884,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=142408: Wed Oct 26 04:42:37 2022
  read: IOPS=302k, BW=148MiB/s (155MB/s)(1033MiB/7001msec)
    slat (nsec): min=1362, max=21871, avg=2810.79, stdev=1238.11
    clat (nsec): min=110, max=126127, avg=10178.08, stdev=2575.13
     lat (usec): min=5, max=127, avg=12.99, stdev= 2.79
    clat percentiles (nsec):
     |  1.00th=[ 6880],  5.00th=[ 7968], 10.00th=[ 8512], 20.00th=[ 9536],
     | 30.00th=[ 9664], 40.00th=[ 9664], 50.00th=[ 9792], 60.00th=[ 9792],
     | 70.00th=[ 9920], 80.00th=[10048], 90.00th=[11456], 95.00th=[11968],
     | 99.00th=[23424], 99.50th=[24960], 99.90th=[30336], 99.95th=[34560],
     | 99.99th=[71168]
   bw (  KiB/s): min=150921, max=151477, per=100.00%, avg=151202.64, stdev=172.05, samples=14
   iops        : min=301842, max=302954, avg=302405.50, stdev=344.06, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.02%, 4=0.11%, 10=79.62%, 20=18.50%, 50=1.70%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=17.59%, sys=80.17%, ctx=12447, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2116575,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1033MiB (1084MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_io_uring-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752171,
  "timestamp_ms" : 1666752171569,
  "time" : "Wed Oct 26 04:42:51 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1086826496,
        "io_kbytes" : 1061354,
        "bw_bytes" : 155238751,
        "bw" : 151600,
        "iops" : 303200.114269,
        "runtime" : 7001,
        "total_ios" : 2122704,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1362,
          "max" : 69741,
          "mean" : 2801.643828,
          "stddev" : 1238.790601,
          "N" : 2122705
        },
        "clat_ns" : {
          "min" : 110,
          "max" : 124213,
          "mean" : 10150.495425,
          "stddev" : 2562.103546,
          "N" : 2122707,
          "percentile" : {
            "1.000000" : 6880,
            "5.000000" : 7968,
            "10.000000" : 9408,
            "20.000000" : 9536,
            "30.000000" : 9664,
            "40.000000" : 9664,
            "50.000000" : 9792,
            "60.000000" : 9792,
            "70.000000" : 9920,
            "80.000000" : 9920,
            "90.000000" : 11328,
            "95.000000" : 11840,
            "99.000000" : 23424,
            "99.500000" : 24960,
            "99.900000" : 30336,
            "99.950000" : 34560,
            "99.990000" : 70144
          }
        },
        "lat_ns" : {
          "min" : 5861,
          "max" : 125596,
          "mean" : 12952.138615,
          "stddev" : 2763.745120,
          "N" : 2122707
        },
        "bw_min" : 151397,
        "bw_max" : 151887,
        "bw_agg" : 100.000000,
        "bw_mean" : 151658.000000,
        "bw_dev" : 143.826864,
        "bw_samples" : 13,
        "iops_min" : 302794,
        "iops_max" : 303775,
        "iops_mean" : 303316.076923,
        "iops_stddev" : 287.786513,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.157143,
      "sys_cpu" : 78.600000,
      "ctx" : 12468,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.049371,
        "4" : 0.088990,
        "10" : 81.786580,
        "20" : 16.352115,
        "50" : 1.670982,
        "100" : 0.044519,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=142447: Wed Oct 26 04:42:51 2022
  read: IOPS=303k, BW=148MiB/s (155MB/s)(1036MiB/7001msec)
    slat (nsec): min=1362, max=69741, avg=2801.64, stdev=1238.79
    clat (nsec): min=110, max=124213, avg=10150.50, stdev=2562.10
     lat (usec): min=5, max=125, avg=12.95, stdev= 2.76
    clat percentiles (nsec):
     |  1.00th=[ 6880],  5.00th=[ 7968], 10.00th=[ 9408], 20.00th=[ 9536],
     | 30.00th=[ 9664], 40.00th=[ 9664], 50.00th=[ 9792], 60.00th=[ 9792],
     | 70.00th=[ 9920], 80.00th=[ 9920], 90.00th=[11328], 95.00th=[11840],
     | 99.00th=[23424], 99.50th=[24960], 99.90th=[30336], 99.95th=[34560],
     | 99.99th=[70144]
   bw (  KiB/s): min=151397, max=151887, per=100.00%, avg=151658.00, stdev=143.83, samples=13
   iops        : min=302794, max=303775, avg=303316.08, stdev=287.79, samples=13
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.05%, 4=0.09%, 10=81.79%, 20=16.35%, 50=1.67%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=19.16%, sys=78.60%, ctx=12468, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2122704,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1036MiB (1087MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-reference_0.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752185,
  "timestamp_ms" : 1666752185221,
  "time" : "Wed Oct 26 04:43:05 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1041431040,
        "io_kbytes" : 1017022,
        "bw_bytes" : 148754612,
        "bw" : 145268,
        "iops" : 290535.780603,
        "runtime" : 7001,
        "total_ios" : 2034041,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1403,
          "max" : 42539,
          "mean" : 2718.299399,
          "stddev" : 1303.395648,
          "N" : 2034042
        },
        "clat_ns" : {
          "min" : 4479,
          "max" : 124985,
          "mean" : 10812.651804,
          "stddev" : 2728.741015,
          "N" : 2034044,
          "percentile" : {
            "1.000000" : 9024,
            "5.000000" : 9024,
            "10.000000" : 9024,
            "20.000000" : 9152,
            "30.000000" : 9280,
            "40.000000" : 10304,
            "50.000000" : 10432,
            "60.000000" : 10560,
            "70.000000" : 11584,
            "80.000000" : 11712,
            "90.000000" : 11840,
            "95.000000" : 12096,
            "99.000000" : 23680,
            "99.500000" : 28800,
            "99.900000" : 35072,
            "99.950000" : 40704,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 8055,
          "max" : 129283,
          "mean" : 13530.950614,
          "stddev" : 2627.157009,
          "N" : 2034044
        },
        "bw_min" : 144981,
        "bw_max" : 145561,
        "bw_agg" : 100.000000,
        "bw_mean" : 145329.000000,
        "bw_dev" : 165.697593,
        "bw_samples" : 14,
        "iops_min" : 289962,
        "iops_max" : 291122,
        "iops_mean" : 290658.071429,
        "iops_stddev" : 331.438465,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 34.800000,
      "sys_cpu" : 63.071429,
      "ctx" : 12160,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 32.978195,
        "20" : 65.542287,
        "50" : 1.435320,
        "100" : 0.043804,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2916402,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 15253,
      "write_ticks" : 0,
      "in_queue" : 15253,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=142487: Wed Oct 26 04:43:05 2022
  read: IOPS=291k, BW=142MiB/s (149MB/s)(993MiB/7001msec)
    slat (nsec): min=1403, max=42539, avg=2718.30, stdev=1303.40
    clat (usec): min=4, max=124, avg=10.81, stdev= 2.73
     lat (usec): min=8, max=129, avg=13.53, stdev= 2.63
    clat percentiles (nsec):
     |  1.00th=[ 9024],  5.00th=[ 9024], 10.00th=[ 9024], 20.00th=[ 9152],
     | 30.00th=[ 9280], 40.00th=[10304], 50.00th=[10432], 60.00th=[10560],
     | 70.00th=[11584], 80.00th=[11712], 90.00th=[11840], 95.00th=[12096],
     | 99.00th=[23680], 99.50th=[28800], 99.90th=[35072], 99.95th=[40704],
     | 99.99th=[72192]
   bw (  KiB/s): min=144981, max=145561, per=100.00%, avg=145329.00, stdev=165.70, samples=14
   iops        : min=289962, max=291122, avg=290658.07, stdev=331.44, samples=14
  lat (usec)   : 10=32.98%, 20=65.54%, 50=1.44%, 100=0.04%, 250=0.01%
  cpu          : usr=34.80%, sys=63.07%, ctx=12160, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2034041,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=993MiB (1041MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2916402/0, merge=0/0, ticks=15253/0, in_queue=15253, util=99.09%

$ rm /tmp/fio-output_512_4_libaio-reference_1.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752198,
  "timestamp_ms" : 1666752198867,
  "time" : "Wed Oct 26 04:43:18 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1038159872,
        "io_kbytes" : 1013828,
        "bw_bytes" : 148287369,
        "bw" : 144811,
        "iops" : 289623.196686,
        "runtime" : 7001,
        "total_ios" : 2027652,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1402,
          "max" : 25027,
          "mean" : 2728.420388,
          "stddev" : 1317.432382,
          "N" : 2027653
        },
        "clat_ns" : {
          "min" : 4689,
          "max" : 93515,
          "mean" : 10845.511402,
          "stddev" : 2729.393787,
          "N" : 2027655,
          "percentile" : {
            "1.000000" : 9024,
            "5.000000" : 9024,
            "10.000000" : 9152,
            "20.000000" : 9152,
            "30.000000" : 9280,
            "40.000000" : 10304,
            "50.000000" : 10432,
            "60.000000" : 10816,
            "70.000000" : 11584,
            "80.000000" : 11712,
            "90.000000" : 11840,
            "95.000000" : 11968,
            "99.000000" : 23424,
            "99.500000" : 29056,
            "99.900000" : 35072,
            "99.950000" : 40192,
            "99.990000" : 71168
          }
        },
        "lat_ns" : {
          "min" : 8115,
          "max" : 98064,
          "mean" : 13573.932355,
          "stddev" : 2591.494403,
          "N" : 2027655
        },
        "bw_min" : 144516,
        "bw_max" : 145113,
        "bw_agg" : 100.000000,
        "bw_mean" : 144877.230769,
        "bw_dev" : 161.265906,
        "bw_samples" : 13,
        "iops_min" : 289032,
        "iops_max" : 290226,
        "iops_mean" : 289754.692308,
        "iops_stddev" : 322.712820,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 34.757143,
      "sys_cpu" : 63.128571,
      "ctx" : 12125,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 36.527964,
        "20" : 61.978979,
        "50" : 1.450989,
        "100" : 0.042216,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2908270,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 15201,
      "write_ticks" : 0,
      "in_queue" : 15201,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=142526: Wed Oct 26 04:43:18 2022
  read: IOPS=290k, BW=141MiB/s (148MB/s)(990MiB/7001msec)
    slat (nsec): min=1402, max=25027, avg=2728.42, stdev=1317.43
    clat (nsec): min=4689, max=93515, avg=10845.51, stdev=2729.39
     lat (nsec): min=8115, max=98064, avg=13573.93, stdev=2591.49
    clat percentiles (nsec):
     |  1.00th=[ 9024],  5.00th=[ 9024], 10.00th=[ 9152], 20.00th=[ 9152],
     | 30.00th=[ 9280], 40.00th=[10304], 50.00th=[10432], 60.00th=[10816],
     | 70.00th=[11584], 80.00th=[11712], 90.00th=[11840], 95.00th=[11968],
     | 99.00th=[23424], 99.50th=[29056], 99.90th=[35072], 99.95th=[40192],
     | 99.99th=[71168]
   bw (  KiB/s): min=144516, max=145113, per=100.00%, avg=144877.23, stdev=161.27, samples=13
   iops        : min=289032, max=290226, avg=289754.69, stdev=322.71, samples=13
  lat (usec)   : 10=36.53%, 20=61.98%, 50=1.45%, 100=0.04%
  cpu          : usr=34.76%, sys=63.13%, ctx=12125, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2027652,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=141MiB/s (148MB/s), 141MiB/s-141MiB/s (148MB/s-148MB/s), io=990MiB (1038MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2908270/0, merge=0/0, ticks=15201/0, in_queue=15201, util=99.09%

$ rm /tmp/fio-output_512_4_libaio-reference_2.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751053,
  "timestamp_ms" : 1666751053526,
  "time" : "Wed Oct 26 04:24:13 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 439462400,
        "io_kbytes" : 429162,
        "bw_bytes" : 62771375,
        "bw" : 61300,
        "iops" : 122600.199971,
        "runtime" : 7001,
        "total_ios" : 858324,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 701,
          "max" : 13997,
          "mean" : 918.099893,
          "stddev" : 66.379994,
          "N" : 858325
        },
        "clat_ns" : {
          "min" : 551,
          "max" : 101871,
          "mean" : 6972.345603,
          "stddev" : 1116.060535,
          "N" : 858324,
          "percentile" : {
            "1.000000" : 6368,
            "5.000000" : 6496,
            "10.000000" : 6560,
            "20.000000" : 6688,
            "30.000000" : 6752,
            "40.000000" : 6752,
            "50.000000" : 6816,
            "60.000000" : 6944,
            "70.000000" : 6944,
            "80.000000" : 7008,
            "90.000000" : 7136,
            "95.000000" : 7264,
            "99.000000" : 12864,
            "99.500000" : 13632,
            "99.900000" : 16512,
            "99.950000" : 16768,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 7013,
          "max" : 102792,
          "mean" : 7890.445503,
          "stddev" : 1119.494292,
          "N" : 858324
        },
        "bw_min" : 61255,
        "bw_max" : 61417,
        "bw_agg" : 100.000000,
        "bw_mean" : 61317.071429,
        "bw_dev" : 43.492642,
        "bw_samples" : 14,
        "iops_min" : 122510,
        "iops_max" : 122834,
        "iops_mean" : 122634.214286,
        "iops_stddev" : 86.918334,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 22,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.301574,
        "20" : 1.679669,
        "50" : 0.010000,
        "100" : 0.014563,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139019: Wed Oct 26 04:24:13 2022
  read: IOPS=123k, BW=59.9MiB/s (62.8MB/s)(419MiB/7001msec)
    slat (nsec): min=701, max=13997, avg=918.10, stdev=66.38
    clat (nsec): min=551, max=101871, avg=6972.35, stdev=1116.06
     lat (usec): min=7, max=102, avg= 7.89, stdev= 1.12
    clat percentiles (nsec):
     |  1.00th=[ 6368],  5.00th=[ 6496], 10.00th=[ 6560], 20.00th=[ 6688],
     | 30.00th=[ 6752], 40.00th=[ 6752], 50.00th=[ 6816], 60.00th=[ 6944],
     | 70.00th=[ 6944], 80.00th=[ 7008], 90.00th=[ 7136], 95.00th=[ 7264],
     | 99.00th=[12864], 99.50th=[13632], 99.90th=[16512], 99.95th=[16768],
     | 99.99th=[58112]
   bw (  KiB/s): min=61255, max=61417, per=100.00%, avg=61317.07, stdev=43.49, samples=14
   iops        : min=122510, max=122834, avg=122634.21, stdev=86.92, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 10=98.30%, 20=1.68%, 50=0.01%, 100=0.01%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=22, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=858324,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=59.9MiB/s (62.8MB/s), 59.9MiB/s-59.9MiB/s (62.8MB/s-62.8MB/s), io=419MiB (439MB), run=7001-7001msec

$ cat /tmp/fio-output_512_4_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752212,
  "timestamp_ms" : 1666752212538,
  "time" : "Wed Oct 26 04:43:32 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1037223424,
        "io_kbytes" : 1012913,
        "bw_bytes" : 148153610,
        "bw" : 144681,
        "iops" : 289361.948293,
        "runtime" : 7001,
        "total_ios" : 2025823,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1412,
          "max" : 28864,
          "mean" : 2730.817650,
          "stddev" : 1328.449286,
          "N" : 2025824
        },
        "clat_ns" : {
          "min" : 4679,
          "max" : 134222,
          "mean" : 10856.180382,
          "stddev" : 2794.572731,
          "N" : 2025826,
          "percentile" : {
            "1.000000" : 9024,
            "5.000000" : 9024,
            "10.000000" : 9152,
            "20.000000" : 9152,
            "30.000000" : 9280,
            "40.000000" : 10304,
            "50.000000" : 10432,
            "60.000000" : 10816,
            "70.000000" : 11584,
            "80.000000" : 11712,
            "90.000000" : 11840,
            "95.000000" : 12096,
            "99.000000" : 23680,
            "99.500000" : 29056,
            "99.900000" : 35584,
            "99.950000" : 42240,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 8135,
          "max" : 137117,
          "mean" : 13586.998051,
          "stddev" : 2658.191293,
          "N" : 2025826
        },
        "bw_min" : 144341,
        "bw_max" : 145442,
        "bw_agg" : 100.000000,
        "bw_mean" : 144773.769231,
        "bw_dev" : 340.786383,
        "bw_samples" : 13,
        "iops_min" : 288682,
        "iops_max" : 290884,
        "iops_mean" : 289547.461538,
        "iops_stddev" : 681.758341,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 35.828571,
      "sys_cpu" : 62.028571,
      "ctx" : 12148,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 36.357915,
        "20" : 62.091604,
        "50" : 1.503537,
        "100" : 0.046253,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2832626,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 14860,
      "write_ticks" : 0,
      "in_queue" : 14860,
      "util" : 99.085923
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=142565: Wed Oct 26 04:43:32 2022
  read: IOPS=289k, BW=141MiB/s (148MB/s)(989MiB/7001msec)
    slat (nsec): min=1412, max=28864, avg=2730.82, stdev=1328.45
    clat (usec): min=4, max=134, avg=10.86, stdev= 2.79
     lat (usec): min=8, max=137, avg=13.59, stdev= 2.66
    clat percentiles (nsec):
     |  1.00th=[ 9024],  5.00th=[ 9024], 10.00th=[ 9152], 20.00th=[ 9152],
     | 30.00th=[ 9280], 40.00th=[10304], 50.00th=[10432], 60.00th=[10816],
     | 70.00th=[11584], 80.00th=[11712], 90.00th=[11840], 95.00th=[12096],
     | 99.00th=[23680], 99.50th=[29056], 99.90th=[35584], 99.95th=[42240],
     | 99.99th=[72192]
   bw (  KiB/s): min=144341, max=145442, per=100.00%, avg=144773.77, stdev=340.79, samples=13
   iops        : min=288682, max=290884, avg=289547.46, stdev=681.76, samples=13
  lat (usec)   : 10=36.36%, 20=62.09%, 50=1.50%, 100=0.05%, 250=0.01%
  cpu          : usr=35.83%, sys=62.03%, ctx=12148, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2025823,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=141MiB/s (148MB/s), 141MiB/s-141MiB/s (148MB/s-148MB/s), io=989MiB (1037MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2832626/0, merge=0/0, ticks=14860/0, in_queue=14860, util=99.09%

$ rm /tmp/fio-output_512_4_libaio-bdev_aio_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752227,
  "timestamp_ms" : 1666752227603,
  "time" : "Wed Oct 26 04:43:47 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1071235584,
        "io_kbytes" : 1046128,
        "bw_bytes" : 153011796,
        "bw" : 149425,
        "iops" : 298850.878446,
        "runtime" : 7001,
        "total_ios" : 2092255,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1583,
          "max" : 29135,
          "mean" : 2702.866753,
          "stddev" : 1007.331736,
          "N" : 2092256
        },
        "clat_ns" : {
          "min" : 260,
          "max" : 114535,
          "mean" : 7822.386945,
          "stddev" : 3118.577266,
          "N" : 2092257,
          "percentile" : {
            "1.000000" : 3280,
            "5.000000" : 3376,
            "10.000000" : 4512,
            "20.000000" : 4832,
            "30.000000" : 6432,
            "40.000000" : 6688,
            "50.000000" : 7200,
            "60.000000" : 8512,
            "70.000000" : 9792,
            "80.000000" : 10048,
            "90.000000" : 11072,
            "95.000000" : 11456,
            "99.000000" : 18560,
            "99.500000" : 21376,
            "99.900000" : 24960,
            "99.950000" : 31872,
            "99.990000" : 63744
          }
        },
        "lat_ns" : {
          "min" : 5791,
          "max" : 116228,
          "mean" : 10525.253804,
          "stddev" : 3235.371221,
          "N" : 2092257
        },
        "bw_min" : 149025,
        "bw_max" : 149805,
        "bw_agg" : 100.000000,
        "bw_mean" : 149469.785714,
        "bw_dev" : 290.944134,
        "bw_samples" : 14,
        "iops_min" : 298050,
        "iops_max" : 299606,
        "iops_mean" : 298939.571429,
        "iops_stddev" : 581.128440,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 43.357143,
      "sys_cpu" : 56.614286,
      "ctx" : 22,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.967043,
        "2" : 60.186784,
        "4" : 38.846173,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.128904,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.015390,
        "4" : 7.186170,
        "10" : 73.121154,
        "20" : 18.881733,
        "50" : 0.629417,
        "100" : 0.036946,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=142613: Wed Oct 26 04:43:47 2022
  read: IOPS=299k, BW=146MiB/s (153MB/s)(1022MiB/7001msec)
    slat (nsec): min=1583, max=29135, avg=2702.87, stdev=1007.33
    clat (nsec): min=260, max=114535, avg=7822.39, stdev=3118.58
     lat (usec): min=5, max=116, avg=10.53, stdev= 3.24
    clat percentiles (nsec):
     |  1.00th=[ 3280],  5.00th=[ 3376], 10.00th=[ 4512], 20.00th=[ 4832],
     | 30.00th=[ 6432], 40.00th=[ 6688], 50.00th=[ 7200], 60.00th=[ 8512],
     | 70.00th=[ 9792], 80.00th=[10048], 90.00th=[11072], 95.00th=[11456],
     | 99.00th=[18560], 99.50th=[21376], 99.90th=[24960], 99.95th=[31872],
     | 99.99th=[63744]
   bw (  KiB/s): min=149025, max=149805, per=100.00%, avg=149469.79, stdev=290.94, samples=14
   iops        : min=298050, max=299606, avg=298939.57, stdev=581.13, samples=14
  lat (nsec)   : 500=0.13%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.02%, 4=7.19%, 10=73.12%, 20=18.88%, 50=0.63%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=43.36%, sys=56.61%, ctx=22, majf=0, minf=0
  IO depths    : 1=1.0%, 2=60.2%, 4=38.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2092255,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=1022MiB (1071MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-bdev_aio_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752242,
  "timestamp_ms" : 1666752242721,
  "time" : "Wed Oct 26 04:44:02 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1061995520,
        "io_kbytes" : 1037105,
        "bw_bytes" : 151691975,
        "bw" : 148136,
        "iops" : 296273.103842,
        "runtime" : 7001,
        "total_ios" : 2074208,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1603,
          "max" : 57037,
          "mean" : 2788.654794,
          "stddev" : 1082.503450,
          "N" : 2074209
        },
        "clat_ns" : {
          "min" : 241,
          "max" : 157215,
          "mean" : 7732.501443,
          "stddev" : 3123.606252,
          "N" : 2074210,
          "percentile" : {
            "1.000000" : 3312,
            "5.000000" : 4512,
            "10.000000" : 4576,
            "20.000000" : 4704,
            "30.000000" : 6432,
            "40.000000" : 6560,
            "50.000000" : 6880,
            "60.000000" : 8256,
            "70.000000" : 9920,
            "80.000000" : 10048,
            "90.000000" : 11200,
            "95.000000" : 11456,
            "99.000000" : 18304,
            "99.500000" : 21120,
            "99.900000" : 25216,
            "99.950000" : 31872,
            "99.990000" : 62720
          }
        },
        "lat_ns" : {
          "min" : 5721,
          "max" : 158908,
          "mean" : 10521.156284,
          "stddev" : 3475.499276,
          "N" : 2074210
        },
        "bw_min" : 147810,
        "bw_max" : 148550,
        "bw_agg" : 100.000000,
        "bw_mean" : 148173.714286,
        "bw_dev" : 244.282443,
        "bw_samples" : 14,
        "iops_min" : 295620,
        "iops_max" : 297100,
        "iops_mean" : 296347.428571,
        "iops_stddev" : 488.564887,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 43.757143,
      "sys_cpu" : 56.214286,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.941564,
        "2" : 61.963410,
        "4" : 37.095026,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.128868,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.017645,
        "4" : 4.562320,
        "10" : 70.359482,
        "20" : 24.313714,
        "50" : 0.580125,
        "100" : 0.036978,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=142662: Wed Oct 26 04:44:02 2022
  read: IOPS=296k, BW=145MiB/s (152MB/s)(1013MiB/7001msec)
    slat (nsec): min=1603, max=57037, avg=2788.65, stdev=1082.50
    clat (nsec): min=241, max=157215, avg=7732.50, stdev=3123.61
     lat (usec): min=5, max=158, avg=10.52, stdev= 3.48
    clat percentiles (nsec):
     |  1.00th=[ 3312],  5.00th=[ 4512], 10.00th=[ 4576], 20.00th=[ 4704],
     | 30.00th=[ 6432], 40.00th=[ 6560], 50.00th=[ 6880], 60.00th=[ 8256],
     | 70.00th=[ 9920], 80.00th=[10048], 90.00th=[11200], 95.00th=[11456],
     | 99.00th=[18304], 99.50th=[21120], 99.90th=[25216], 99.95th=[31872],
     | 99.99th=[62720]
   bw (  KiB/s): min=147810, max=148550, per=100.00%, avg=148173.71, stdev=244.28, samples=14
   iops        : min=295620, max=297100, avg=296347.43, stdev=488.56, samples=14
  lat (nsec)   : 250=0.01%, 500=0.13%, 750=0.01%
  lat (usec)   : 2=0.02%, 4=4.56%, 10=70.36%, 20=24.31%, 50=0.58%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=43.76%, sys=56.21%, ctx=23, majf=0, minf=0
  IO depths    : 1=0.9%, 2=62.0%, 4=37.1%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2074208,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1013MiB (1062MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-bdev_aio_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752257,
  "timestamp_ms" : 1666752257808,
  "time" : "Wed Oct 26 04:44:17 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1064985600,
        "io_kbytes" : 1040025,
        "bw_bytes" : 152119068,
        "bw" : 148553,
        "iops" : 297107.270390,
        "runtime" : 7001,
        "total_ios" : 2080048,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1593,
          "max" : 32571,
          "mean" : 2739.548837,
          "stddev" : 1047.822313,
          "N" : 2080049
        },
        "clat_ns" : {
          "min" : 270,
          "max" : 117951,
          "mean" : 7801.096301,
          "stddev" : 3165.549384,
          "N" : 2080050,
          "percentile" : {
            "1.000000" : 3312,
            "5.000000" : 3440,
            "10.000000" : 4576,
            "20.000000" : 4704,
            "30.000000" : 6432,
            "40.000000" : 6560,
            "50.000000" : 6944,
            "60.000000" : 8512,
            "70.000000" : 9920,
            "80.000000" : 10048,
            "90.000000" : 11200,
            "95.000000" : 11456,
            "99.000000" : 18304,
            "99.500000" : 21120,
            "99.900000" : 25472,
            "99.950000" : 32128,
            "99.990000" : 64256
          }
        },
        "lat_ns" : {
          "min" : 5830,
          "max" : 119664,
          "mean" : 10540.645231,
          "stddev" : 3389.356523,
          "N" : 2080050
        },
        "bw_min" : 148084,
        "bw_max" : 149034,
        "bw_agg" : 100.000000,
        "bw_mean" : 148592.357143,
        "bw_dev" : 285.577043,
        "bw_samples" : 14,
        "iops_min" : 296168,
        "iops_max" : 298068,
        "iops_mean" : 297184.714286,
        "iops_stddev" : 571.154085,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 41.985714,
      "sys_cpu" : 57.985714,
      "ctx" : 25,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.959641,
        "2" : 61.257433,
        "4" : 37.782926,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.123459,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.015096,
        "4" : 6.310912,
        "10" : 69.856369,
        "20" : 23.058602,
        "50" : 0.596188,
        "100" : 0.038797,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=142712: Wed Oct 26 04:44:17 2022
  read: IOPS=297k, BW=145MiB/s (152MB/s)(1016MiB/7001msec)
    slat (nsec): min=1593, max=32571, avg=2739.55, stdev=1047.82
    clat (nsec): min=270, max=117951, avg=7801.10, stdev=3165.55
     lat (usec): min=5, max=119, avg=10.54, stdev= 3.39
    clat percentiles (nsec):
     |  1.00th=[ 3312],  5.00th=[ 3440], 10.00th=[ 4576], 20.00th=[ 4704],
     | 30.00th=[ 6432], 40.00th=[ 6560], 50.00th=[ 6944], 60.00th=[ 8512],
     | 70.00th=[ 9920], 80.00th=[10048], 90.00th=[11200], 95.00th=[11456],
     | 99.00th=[18304], 99.50th=[21120], 99.90th=[25472], 99.95th=[32128],
     | 99.99th=[64256]
   bw (  KiB/s): min=148084, max=149034, per=100.00%, avg=148592.36, stdev=285.58, samples=14
   iops        : min=296168, max=298068, avg=297184.71, stdev=571.15, samples=14
  lat (nsec)   : 500=0.12%
  lat (usec)   : 2=0.02%, 4=6.31%, 10=69.86%, 20=23.06%, 50=0.60%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=41.99%, sys=57.99%, ctx=25, majf=0, minf=0
  IO depths    : 1=1.0%, 2=61.3%, 4=37.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2080048,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=1016MiB (1065MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-reference_0.txt
$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752273,
  "timestamp_ms" : 1666752273038,
  "time" : "Wed Oct 26 04:44:33 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1012254720,
        "io_kbytes" : 988530,
        "bw_bytes" : 144587161,
        "bw" : 141198,
        "iops" : 282396.514784,
        "runtime" : 7001,
        "total_ios" : 1977058,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1603,
          "max" : 63850,
          "mean" : 2849.242207,
          "stddev" : 1275.330036,
          "N" : 1977059
        },
        "clat_ns" : {
          "min" : 641,
          "max" : 100899,
          "mean" : 8513.515201,
          "stddev" : 2759.933150,
          "N" : 1977060,
          "percentile" : {
            "1.000000" : 4832,
            "5.000000" : 4960,
            "10.000000" : 5088,
            "20.000000" : 6880,
            "30.000000" : 7456,
            "40.000000" : 7584,
            "50.000000" : 7648,
            "60.000000" : 9280,
            "70.000000" : 9408,
            "80.000000" : 9536,
            "90.000000" : 11328,
            "95.000000" : 12224,
            "99.000000" : 18560,
            "99.500000" : 21120,
            "99.900000" : 25216,
            "99.950000" : 31872,
            "99.990000" : 63744
          }
        },
        "lat_ns" : {
          "min" : 6052,
          "max" : 102622,
          "mean" : 11362.757426,
          "stddev" : 3379.702263,
          "N" : 1977060
        },
        "bw_min" : 140715,
        "bw_max" : 141758,
        "bw_agg" : 100.000000,
        "bw_mean" : 141236.428571,
        "bw_dev" : 312.103216,
        "bw_samples" : 14,
        "iops_min" : 281430,
        "iops_max" : 283516,
        "iops_mean" : 282472.928571,
        "iops_stddev" : 624.256415,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 43.642857,
      "sys_cpu" : 56.328571,
      "ctx" : 24,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.877010,
        "2" : 56.256974,
        "4" : 42.866016,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.053918,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.015680,
        "4" : 0.791783,
        "10" : 82.251962,
        "20" : 16.201093,
        "50" : 0.643279,
        "100" : 0.037986,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142761: Wed Oct 26 04:44:33 2022
  read: IOPS=282k, BW=138MiB/s (145MB/s)(965MiB/7001msec)
    slat (nsec): min=1603, max=63850, avg=2849.24, stdev=1275.33
    clat (nsec): min=641, max=100899, avg=8513.52, stdev=2759.93
     lat (usec): min=6, max=102, avg=11.36, stdev= 3.38
    clat percentiles (nsec):
     |  1.00th=[ 4832],  5.00th=[ 4960], 10.00th=[ 5088], 20.00th=[ 6880],
     | 30.00th=[ 7456], 40.00th=[ 7584], 50.00th=[ 7648], 60.00th=[ 9280],
     | 70.00th=[ 9408], 80.00th=[ 9536], 90.00th=[11328], 95.00th=[12224],
     | 99.00th=[18560], 99.50th=[21120], 99.90th=[25216], 99.95th=[31872],
     | 99.99th=[63744]
   bw (  KiB/s): min=140715, max=141758, per=100.00%, avg=141236.43, stdev=312.10, samples=14
   iops        : min=281430, max=283516, avg=282472.93, stdev=624.26, samples=14
  lat (nsec)   : 750=0.05%, 1000=0.01%
  lat (usec)   : 2=0.02%, 4=0.79%, 10=82.25%, 20=16.20%, 50=0.64%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=43.64%, sys=56.33%, ctx=24, majf=0, minf=0
  IO depths    : 1=0.9%, 2=56.3%, 4=42.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1977058,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=965MiB (1012MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752288,
  "timestamp_ms" : 1666752288178,
  "time" : "Wed Oct 26 04:44:48 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1009187328,
        "io_kbytes" : 985534,
        "bw_bytes" : 144149025,
        "bw" : 140770,
        "iops" : 281540.494215,
        "runtime" : 7001,
        "total_ios" : 1971065,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1613,
          "max" : 36499,
          "mean" : 2864.161118,
          "stddev" : 1277.898171,
          "N" : 1971066
        },
        "clat_ns" : {
          "min" : 641,
          "max" : 123301,
          "mean" : 8534.661721,
          "stddev" : 2698.421230,
          "N" : 1971065,
          "percentile" : {
            "1.000000" : 4896,
            "5.000000" : 5024,
            "10.000000" : 5152,
            "20.000000" : 6944,
            "30.000000" : 7456,
            "40.000000" : 7584,
            "50.000000" : 7712,
            "60.000000" : 9280,
            "70.000000" : 9408,
            "80.000000" : 9536,
            "90.000000" : 10944,
            "95.000000" : 12224,
            "99.000000" : 18304,
            "99.500000" : 21120,
            "99.900000" : 25472,
            "99.950000" : 32384,
            "99.990000" : 63744
          }
        },
        "lat_ns" : {
          "min" : 6042,
          "max" : 127870,
          "mean" : 11398.823449,
          "stddev" : 3337.911286,
          "N" : 1971065
        },
        "bw_min" : 139935,
        "bw_max" : 141743,
        "bw_agg" : 100.000000,
        "bw_mean" : 140813.428571,
        "bw_dev" : 451.996207,
        "bw_samples" : 14,
        "iops_min" : 279870,
        "iops_max" : 283486,
        "iops_mean" : 281626.928571,
        "iops_stddev" : 904.069642,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 44.785714,
      "sys_cpu" : 55.185714,
      "ctx" : 22,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.869327,
        "2" : 55.711506,
        "4" : 43.419167,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.051191,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.016691,
        "4" : 0.692570,
        "10" : 83.599323,
        "20" : 14.998186,
        "50" : 0.595871,
        "100" : 0.041500,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142811: Wed Oct 26 04:44:48 2022
  read: IOPS=282k, BW=137MiB/s (144MB/s)(962MiB/7001msec)
    slat (nsec): min=1613, max=36499, avg=2864.16, stdev=1277.90
    clat (nsec): min=641, max=123301, avg=8534.66, stdev=2698.42
     lat (usec): min=6, max=127, avg=11.40, stdev= 3.34
    clat percentiles (nsec):
     |  1.00th=[ 4896],  5.00th=[ 5024], 10.00th=[ 5152], 20.00th=[ 6944],
     | 30.00th=[ 7456], 40.00th=[ 7584], 50.00th=[ 7712], 60.00th=[ 9280],
     | 70.00th=[ 9408], 80.00th=[ 9536], 90.00th=[10944], 95.00th=[12224],
     | 99.00th=[18304], 99.50th=[21120], 99.90th=[25472], 99.95th=[32384],
     | 99.99th=[63744]
   bw (  KiB/s): min=139935, max=141743, per=100.00%, avg=140813.43, stdev=452.00, samples=14
   iops        : min=279870, max=283486, avg=281626.93, stdev=904.07, samples=14
  lat (nsec)   : 750=0.05%, 1000=0.01%
  lat (usec)   : 2=0.02%, 4=0.69%, 10=83.60%, 20=15.00%, 50=0.60%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=44.79%, sys=55.19%, ctx=22, majf=0, minf=0
  IO depths    : 1=0.9%, 2=55.7%, 4=43.4%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1971065,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=962MiB (1009MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_4_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752303,
  "timestamp_ms" : 1666752303292,
  "time" : "Wed Oct 26 04:45:03 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1011238912,
        "io_kbytes" : 987538,
        "bw_bytes" : 144442067,
        "bw" : 141056,
        "iops" : 282112.841023,
        "runtime" : 7001,
        "total_ios" : 1975072,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1613,
          "max" : 61526,
          "mean" : 2852.995997,
          "stddev" : 1262.832611,
          "N" : 1975073
        },
        "clat_ns" : {
          "min" : 631,
          "max" : 118362,
          "mean" : 8518.367863,
          "stddev" : 2664.843579,
          "N" : 1975072,
          "percentile" : {
            "1.000000" : 4832,
            "5.000000" : 4960,
            "10.000000" : 5088,
            "20.000000" : 6880,
            "30.000000" : 7456,
            "40.000000" : 7584,
            "50.000000" : 7712,
            "60.000000" : 9280,
            "70.000000" : 9408,
            "80.000000" : 9536,
            "90.000000" : 10944,
            "95.000000" : 12224,
            "99.000000" : 18304,
            "99.500000" : 20864,
            "99.900000" : 24960,
            "99.950000" : 30592,
            "99.990000" : 62208
          }
        },
        "lat_ns" : {
          "min" : 6021,
          "max" : 120075,
          "mean" : 11371.363823,
          "stddev" : 3297.605000,
          "N" : 1975072
        },
        "bw_min" : 140440,
        "bw_max" : 142061,
        "bw_agg" : 100.000000,
        "bw_mean" : 141095.714286,
        "bw_dev" : 440.312377,
        "bw_samples" : 14,
        "iops_min" : 280880,
        "iops_max" : 284122,
        "iops_mean" : 282191.428571,
        "iops_stddev" : 880.624753,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 43.471429,
      "sys_cpu" : 56.500000,
      "ctx" : 20,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.861437,
        "2" : 55.847635,
        "4" : 43.290928,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.052150,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.018075,
        "4" : 0.862348,
        "10" : 82.898092,
        "20" : 15.549256,
        "50" : 0.580789,
        "100" : 0.035391,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=142860: Wed Oct 26 04:45:03 2022
  read: IOPS=282k, BW=138MiB/s (144MB/s)(964MiB/7001msec)
    slat (nsec): min=1613, max=61526, avg=2853.00, stdev=1262.83
    clat (nsec): min=631, max=118362, avg=8518.37, stdev=2664.84
     lat (usec): min=6, max=120, avg=11.37, stdev= 3.30
    clat percentiles (nsec):
     |  1.00th=[ 4832],  5.00th=[ 4960], 10.00th=[ 5088], 20.00th=[ 6880],
     | 30.00th=[ 7456], 40.00th=[ 7584], 50.00th=[ 7712], 60.00th=[ 9280],
     | 70.00th=[ 9408], 80.00th=[ 9536], 90.00th=[10944], 95.00th=[12224],
     | 99.00th=[18304], 99.50th=[20864], 99.90th=[24960], 99.95th=[30592],
     | 99.99th=[62208]
   bw (  KiB/s): min=140440, max=142061, per=100.00%, avg=141095.71, stdev=440.31, samples=14
   iops        : min=280880, max=284122, avg=282191.43, stdev=880.62, samples=14
  lat (nsec)   : 750=0.05%, 1000=0.01%
  lat (usec)   : 2=0.02%, 4=0.86%, 10=82.90%, 20=15.55%, 50=0.58%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=43.47%, sys=56.50%, ctx=20, majf=0, minf=0
  IO depths    : 1=0.9%, 2=55.8%, 4=43.3%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1975072,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=138MiB/s (144MB/s), 138MiB/s-138MiB/s (144MB/s-144MB/s), io=964MiB (1011MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_0.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752317,
  "timestamp_ms" : 1666752317061,
  "time" : "Wed Oct 26 04:45:17 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 995397120,
        "io_kbytes" : 972067,
        "bw_bytes" : 142179277,
        "bw" : 138846,
        "iops" : 277693.329524,
        "runtime" : 7001,
        "total_ios" : 1944131,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1472,
          "max" : 23805,
          "mean" : 2275.900375,
          "stddev" : 900.413544,
          "N" : 1944132
        },
        "clat_ns" : {
          "min" : 4949,
          "max" : 133701,
          "mean" : 11331.964851,
          "stddev" : 2845.799578,
          "N" : 1944134,
          "percentile" : {
            "1.000000" : 9408,
            "5.000000" : 9536,
            "10.000000" : 9536,
            "20.000000" : 9664,
            "30.000000" : 9664,
            "40.000000" : 9792,
            "50.000000" : 11072,
            "60.000000" : 12096,
            "70.000000" : 12224,
            "80.000000" : 12352,
            "90.000000" : 12352,
            "95.000000" : 12608,
            "99.000000" : 23680,
            "99.500000" : 28288,
            "99.900000" : 36096,
            "99.950000" : 40192,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 7484,
          "max" : 136586,
          "mean" : 13607.865899,
          "stddev" : 2632.972492,
          "N" : 1944134
        },
        "bw_min" : 138554,
        "bw_max" : 139440,
        "bw_agg" : 100.000000,
        "bw_mean" : 138885.857143,
        "bw_dev" : 275.370200,
        "bw_samples" : 14,
        "iops_min" : 277108,
        "iops_max" : 278880,
        "iops_mean" : 277771.785714,
        "iops_stddev" : 550.705586,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 36.685714,
      "sys_cpu" : 61.342857,
      "ctx" : 11813,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 45.621154,
        "20" : 52.590952,
        "50" : 1.747002,
        "100" : 0.039349,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=142899: Wed Oct 26 04:45:17 2022
  read: IOPS=278k, BW=136MiB/s (142MB/s)(949MiB/7001msec)
    slat (nsec): min=1472, max=23805, avg=2275.90, stdev=900.41
    clat (usec): min=4, max=133, avg=11.33, stdev= 2.85
     lat (usec): min=7, max=136, avg=13.61, stdev= 2.63
    clat percentiles (nsec):
     |  1.00th=[ 9408],  5.00th=[ 9536], 10.00th=[ 9536], 20.00th=[ 9664],
     | 30.00th=[ 9664], 40.00th=[ 9792], 50.00th=[11072], 60.00th=[12096],
     | 70.00th=[12224], 80.00th=[12352], 90.00th=[12352], 95.00th=[12608],
     | 99.00th=[23680], 99.50th=[28288], 99.90th=[36096], 99.95th=[40192],
     | 99.99th=[72192]
   bw (  KiB/s): min=138554, max=139440, per=100.00%, avg=138885.86, stdev=275.37, samples=14
   iops        : min=277108, max=278880, avg=277771.79, stdev=550.71, samples=14
  lat (usec)   : 10=45.62%, 20=52.59%, 50=1.75%, 100=0.04%, 250=0.01%
  cpu          : usr=36.69%, sys=61.34%, ctx=11813, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1944131,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=136MiB/s (142MB/s), 136MiB/s-136MiB/s (142MB/s-142MB/s), io=949MiB (995MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752330,
  "timestamp_ms" : 1666752330841,
  "time" : "Wed Oct 26 04:45:30 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 998190080,
        "io_kbytes" : 974795,
        "bw_bytes" : 142578214,
        "bw" : 139236,
        "iops" : 278472.503928,
        "runtime" : 7001,
        "total_ios" : 1949586,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1462,
          "max" : 34996,
          "mean" : 2286.772988,
          "stddev" : 911.175884,
          "N" : 1949587
        },
        "clat_ns" : {
          "min" : 501,
          "max" : 134612,
          "mean" : 11299.563884,
          "stddev" : 2830.245084,
          "N" : 1949589,
          "percentile" : {
            "1.000000" : 9408,
            "5.000000" : 9536,
            "10.000000" : 9536,
            "20.000000" : 9536,
            "30.000000" : 9664,
            "40.000000" : 9664,
            "50.000000" : 10944,
            "60.000000" : 12096,
            "70.000000" : 12224,
            "80.000000" : 12224,
            "90.000000" : 12352,
            "95.000000" : 12608,
            "99.000000" : 23680,
            "99.500000" : 28288,
            "99.900000" : 36096,
            "99.950000" : 41216,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 8206,
          "max" : 138840,
          "mean" : 13586.337374,
          "stddev" : 2615.938007,
          "N" : 1949589
        },
        "bw_min" : 138799,
        "bw_max" : 139789,
        "bw_agg" : 100.000000,
        "bw_mean" : 139298.538462,
        "bw_dev" : 326.805145,
        "bw_samples" : 13,
        "iops_min" : 277600,
        "iops_max" : 279578,
        "iops_mean" : 278597.230769,
        "iops_stddev" : 653.355717,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 38.457143,
      "sys_cpu" : 59.557143,
      "ctx" : 11881,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 45.654565,
        "20" : 52.723501,
        "50" : 1.577668,
        "100" : 0.043599,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=142939: Wed Oct 26 04:45:30 2022
  read: IOPS=278k, BW=136MiB/s (143MB/s)(952MiB/7001msec)
    slat (nsec): min=1462, max=34996, avg=2286.77, stdev=911.18
    clat (nsec): min=501, max=134612, avg=11299.56, stdev=2830.25
     lat (usec): min=8, max=138, avg=13.59, stdev= 2.62
    clat percentiles (nsec):
     |  1.00th=[ 9408],  5.00th=[ 9536], 10.00th=[ 9536], 20.00th=[ 9536],
     | 30.00th=[ 9664], 40.00th=[ 9664], 50.00th=[10944], 60.00th=[12096],
     | 70.00th=[12224], 80.00th=[12224], 90.00th=[12352], 95.00th=[12608],
     | 99.00th=[23680], 99.50th=[28288], 99.90th=[36096], 99.95th=[41216],
     | 99.99th=[72192]
   bw (  KiB/s): min=138799, max=139789, per=100.00%, avg=139298.54, stdev=326.81, samples=13
   iops        : min=277600, max=279578, avg=278597.23, stdev=653.36, samples=13
  lat (nsec)   : 750=0.01%
  lat (usec)   : 10=45.65%, 20=52.72%, 50=1.58%, 100=0.04%, 250=0.01%
  cpu          : usr=38.46%, sys=59.56%, ctx=11881, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1949586,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=952MiB (998MB), run=7001-7001msec

$ rm /tmp/fio-output_512_4_libaio-xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_4_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_4_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752344,
  "timestamp_ms" : 1666752344512,
  "time" : "Wed Oct 26 04:45:44 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1003818496,
        "io_kbytes" : 980291,
        "bw_bytes" : 143382159,
        "bw" : 140021,
        "iops" : 280042.708185,
        "runtime" : 7001,
        "total_ios" : 1960579,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1462,
          "max" : 27762,
          "mean" : 2373.694871,
          "stddev" : 1047.956777,
          "N" : 1960580
        },
        "clat_ns" : {
          "min" : 4919,
          "max" : 134282,
          "mean" : 11239.872907,
          "stddev" : 2798.284062,
          "N" : 1960582,
          "percentile" : {
            "1.000000" : 9408,
            "5.000000" : 9408,
            "10.000000" : 9408,
            "20.000000" : 9536,
            "30.000000" : 9536,
            "40.000000" : 9664,
            "50.000000" : 10944,
            "60.000000" : 11968,
            "70.000000" : 12096,
            "80.000000" : 12224,
            "90.000000" : 12352,
            "95.000000" : 12608,
            "99.000000" : 23680,
            "99.500000" : 28288,
            "99.900000" : 36096,
            "99.950000" : 40192,
            "99.990000" : 70144
          }
        },
        "lat_ns" : {
          "min" : 7174,
          "max" : 138400,
          "mean" : 13613.568208,
          "stddev" : 2576.989435,
          "N" : 1960582
        },
        "bw_min" : 139649,
        "bw_max" : 140398,
        "bw_agg" : 100.000000,
        "bw_mean" : 140066.571429,
        "bw_dev" : 220.341593,
        "bw_samples" : 14,
        "iops_min" : 279298,
        "iops_max" : 280796,
        "iops_mean" : 280133.214286,
        "iops_stddev" : 440.635935,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 38.142857,
      "sys_cpu" : 59.885714,
      "ctx" : 11851,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 45.917150,
        "20" : 52.538204,
        "50" : 1.504913,
        "100" : 0.039019,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=142978: Wed Oct 26 04:45:44 2022
  read: IOPS=280k, BW=137MiB/s (143MB/s)(957MiB/7001msec)
    slat (nsec): min=1462, max=27762, avg=2373.69, stdev=1047.96
    clat (usec): min=4, max=134, avg=11.24, stdev= 2.80
     lat (usec): min=7, max=138, avg=13.61, stdev= 2.58
    clat percentiles (nsec):
     |  1.00th=[ 9408],  5.00th=[ 9408], 10.00th=[ 9408], 20.00th=[ 9536],
     | 30.00th=[ 9536], 40.00th=[ 9664], 50.00th=[10944], 60.00th=[11968],
     | 70.00th=[12096], 80.00th=[12224], 90.00th=[12352], 95.00th=[12608],
     | 99.00th=[23680], 99.50th=[28288], 99.90th=[36096], 99.95th=[40192],
     | 99.99th=[70144]
   bw (  KiB/s): min=139649, max=140398, per=100.00%, avg=140066.57, stdev=220.34, samples=14
   iops        : min=279298, max=280796, avg=280133.21, stdev=440.64, samples=14
  lat (usec)   : 10=45.92%, 20=52.54%, 50=1.50%, 100=0.04%, 250=0.01%
  cpu          : usr=38.14%, sys=59.89%, ctx=11851, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1960579,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=957MiB (1004MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751067,
  "timestamp_ms" : 1666751067207,
  "time" : "Wed Oct 26 04:24:27 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 282276352,
        "io_kbytes" : 275660,
        "bw_bytes" : 40319433,
        "bw" : 39374,
        "iops" : 78748.893015,
        "runtime" : 7001,
        "total_ios" : 551321,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2504,
          "max" : 49713,
          "mean" : 2754.340312,
          "stddev" : 211.568405,
          "N" : 551321
        },
        "clat_ns" : {
          "min" : 200,
          "max" : 99306,
          "mean" : 9273.575117,
          "stddev" : 1277.471589,
          "N" : 551321,
          "percentile" : {
            "1.000000" : 8896,
            "5.000000" : 8896,
            "10.000000" : 9024,
            "20.000000" : 9024,
            "30.000000" : 9024,
            "40.000000" : 9024,
            "50.000000" : 9152,
            "60.000000" : 9152,
            "70.000000" : 9152,
            "80.000000" : 9152,
            "90.000000" : 9280,
            "95.000000" : 9536,
            "99.000000" : 15168,
            "99.500000" : 15808,
            "99.900000" : 19840,
            "99.950000" : 26240,
            "99.990000" : 56064
          }
        },
        "lat_ns" : {
          "min" : 9197,
          "max" : 102102,
          "mean" : 12027.915429,
          "stddev" : 1296.149259,
          "N" : 551321
        },
        "bw_min" : 39234,
        "bw_max" : 39603,
        "bw_agg" : 100.000000,
        "bw_mean" : 39390.142857,
        "bw_dev" : 106.081142,
        "bw_samples" : 14,
        "iops_min" : 78468,
        "iops_max" : 79206,
        "iops_mean" : 78780.357143,
        "iops_stddev" : 212.094176,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.414286,
      "sys_cpu" : 48.628571,
      "ctx" : 551215,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 97.031312,
        "20" : 2.883982,
        "50" : 0.057317,
        "100" : 0.019408,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 793290,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5015,
      "write_ticks" : 0,
      "in_queue" : 5015,
      "util" : 99.049128
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=139058: Wed Oct 26 04:24:27 2022
  read: IOPS=78.7k, BW=38.5MiB/s (40.3MB/s)(269MiB/7001msec)
    slat (nsec): min=2504, max=49713, avg=2754.34, stdev=211.57
    clat (nsec): min=200, max=99306, avg=9273.58, stdev=1277.47
     lat (usec): min=9, max=102, avg=12.03, stdev= 1.30
    clat percentiles (nsec):
     |  1.00th=[ 8896],  5.00th=[ 8896], 10.00th=[ 9024], 20.00th=[ 9024],
     | 30.00th=[ 9024], 40.00th=[ 9024], 50.00th=[ 9152], 60.00th=[ 9152],
     | 70.00th=[ 9152], 80.00th=[ 9152], 90.00th=[ 9280], 95.00th=[ 9536],
     | 99.00th=[15168], 99.50th=[15808], 99.90th=[19840], 99.95th=[26240],
     | 99.99th=[56064]
   bw (  KiB/s): min=39234, max=39603, per=100.00%, avg=39390.14, stdev=106.08, samples=14
   iops        : min=78468, max=79206, avg=78780.36, stdev=212.09, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 10=97.03%, 20=2.88%, 50=0.06%, 100=0.02%
  cpu          : usr=12.41%, sys=48.63%, ctx=551215, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=551321,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=38.5MiB/s (40.3MB/s), 38.5MiB/s-38.5MiB/s (40.3MB/s-40.3MB/s), io=269MiB (282MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=793290/0, merge=0/0, ticks=5015/0, in_queue=5015, util=99.05%

$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752358,
  "timestamp_ms" : 1666752358142,
  "time" : "Wed Oct 26 04:45:58 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1195644416,
        "io_kbytes" : 1167621,
        "bw_bytes" : 170781947,
        "bw" : 166779,
        "iops" : 333557.491787,
        "runtime" : 7001,
        "total_ios" : 2335236,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1152,
          "max" : 19917,
          "mean" : 2244.668577,
          "stddev" : 1068.788020,
          "N" : 2335236
        },
        "clat_ns" : {
          "min" : 80,
          "max" : 145854,
          "mean" : 21249.490787,
          "stddev" : 3863.814320,
          "N" : 2335243,
          "percentile" : {
            "1.000000" : 13760,
            "5.000000" : 18816,
            "10.000000" : 19072,
            "20.000000" : 19328,
            "30.000000" : 20352,
            "40.000000" : 20608,
            "50.000000" : 20608,
            "60.000000" : 20864,
            "70.000000" : 21376,
            "80.000000" : 21888,
            "90.000000" : 22656,
            "95.000000" : 24960,
            "99.000000" : 41728,
            "99.500000" : 44288,
            "99.900000" : 56064,
            "99.950000" : 67072,
            "99.990000" : 82432
          }
        },
        "lat_ns" : {
          "min" : 6612,
          "max" : 147096,
          "mean" : 23494.157402,
          "stddev" : 3821.091989,
          "N" : 2335243
        },
        "bw_min" : 166104,
        "bw_max" : 167290,
        "bw_agg" : 100.000000,
        "bw_mean" : 166794.153846,
        "bw_dev" : 286.537329,
        "bw_samples" : 13,
        "iops_min" : 332210,
        "iops_max" : 334580,
        "iops_mean" : 333588.461538,
        "iops_stddev" : 572.567989,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.142857,
      "sys_cpu" : 83.414286,
      "ctx" : 12659,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999957,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.016915,
        "20" : 22.109457,
        "50" : 77.740494,
        "100" : 0.130993,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=143018: Wed Oct 26 04:45:58 2022
  read: IOPS=334k, BW=163MiB/s (171MB/s)(1140MiB/7001msec)
    slat (nsec): min=1152, max=19917, avg=2244.67, stdev=1068.79
    clat (nsec): min=80, max=145854, avg=21249.49, stdev=3863.81
     lat (usec): min=6, max=147, avg=23.49, stdev= 3.82
    clat percentiles (nsec):
     |  1.00th=[13760],  5.00th=[18816], 10.00th=[19072], 20.00th=[19328],
     | 30.00th=[20352], 40.00th=[20608], 50.00th=[20608], 60.00th=[20864],
     | 70.00th=[21376], 80.00th=[21888], 90.00th=[22656], 95.00th=[24960],
     | 99.00th=[41728], 99.50th=[44288], 99.90th=[56064], 99.95th=[67072],
     | 99.99th=[82432]
   bw (  KiB/s): min=166104, max=167290, per=100.00%, avg=166794.15, stdev=286.54, samples=13
   iops        : min=332210, max=334580, avg=333588.46, stdev=572.57, samples=13
  lat (nsec)   : 100=0.01%
  lat (usec)   : 4=0.01%, 10=0.02%, 20=22.11%, 50=77.74%, 100=0.13%
  lat (usec)   : 250=0.01%
  cpu          : usr=14.14%, sys=83.41%, ctx=12659, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2335236,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=1140MiB (1196MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752371,
  "timestamp_ms" : 1666752371752,
  "time" : "Wed Oct 26 04:46:11 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1196563968,
        "io_kbytes" : 1168519,
        "bw_bytes" : 170913293,
        "bw" : 166907,
        "iops" : 333813.883731,
        "runtime" : 7001,
        "total_ios" : 2337031,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1152,
          "max" : 24005,
          "mean" : 2220.236975,
          "stddev" : 1090.837490,
          "N" : 2337032
        },
        "clat_ns" : {
          "min" : 3196,
          "max" : 144360,
          "mean" : 21244.316029,
          "stddev" : 3839.831790,
          "N" : 2337038,
          "percentile" : {
            "1.000000" : 13888,
            "5.000000" : 18816,
            "10.000000" : 19072,
            "20.000000" : 19328,
            "30.000000" : 20352,
            "40.000000" : 20608,
            "50.000000" : 20608,
            "60.000000" : 20864,
            "70.000000" : 21632,
            "80.000000" : 21888,
            "90.000000" : 22912,
            "95.000000" : 24960,
            "99.000000" : 41728,
            "99.500000" : 44288,
            "99.900000" : 53504,
            "99.950000" : 64768,
            "99.990000" : 81408
          }
        },
        "lat_ns" : {
          "min" : 7093,
          "max" : 145563,
          "mean" : 23464.552185,
          "stddev" : 3807.683431,
          "N" : 2337038
        },
        "bw_min" : 166713,
        "bw_max" : 167275,
        "bw_agg" : 100.000000,
        "bw_mean" : 166953.357143,
        "bw_dev" : 172.887517,
        "bw_samples" : 14,
        "iops_min" : 333426,
        "iops_max" : 334550,
        "iops_mean" : 333906.928571,
        "iops_stddev" : 345.829763,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.800000,
      "sys_cpu" : 83.771429,
      "ctx" : 12624,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999957,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.015575,
        "20" : 23.539739,
        "50" : 76.322522,
        "100" : 0.120965,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=143058: Wed Oct 26 04:46:11 2022
  read: IOPS=334k, BW=163MiB/s (171MB/s)(1141MiB/7001msec)
    slat (nsec): min=1152, max=24005, avg=2220.24, stdev=1090.84
    clat (usec): min=3, max=144, avg=21.24, stdev= 3.84
     lat (usec): min=7, max=145, avg=23.46, stdev= 3.81
    clat percentiles (nsec):
     |  1.00th=[13888],  5.00th=[18816], 10.00th=[19072], 20.00th=[19328],
     | 30.00th=[20352], 40.00th=[20608], 50.00th=[20608], 60.00th=[20864],
     | 70.00th=[21632], 80.00th=[21888], 90.00th=[22912], 95.00th=[24960],
     | 99.00th=[41728], 99.50th=[44288], 99.90th=[53504], 99.95th=[64768],
     | 99.99th=[81408]
   bw (  KiB/s): min=166713, max=167275, per=100.00%, avg=166953.36, stdev=172.89, samples=14
   iops        : min=333426, max=334550, avg=333906.93, stdev=345.83, samples=14
  lat (usec)   : 4=0.01%, 10=0.02%, 20=23.54%, 50=76.32%, 100=0.12%
  lat (usec)   : 250=0.01%
  cpu          : usr=13.80%, sys=83.77%, ctx=12624, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2337031,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=1141MiB (1197MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752385,
  "timestamp_ms" : 1666752385359,
  "time" : "Wed Oct 26 04:46:25 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1210347520,
        "io_kbytes" : 1181980,
        "bw_bytes" : 172882091,
        "bw" : 168830,
        "iops" : 337659.334381,
        "runtime" : 7001,
        "total_ios" : 2363953,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1122,
          "max" : 22653,
          "mean" : 2034.197053,
          "stddev" : 1011.133446,
          "N" : 2363953
        },
        "clat_ns" : {
          "min" : 90,
          "max" : 142347,
          "mean" : 21168.681512,
          "stddev" : 3823.819840,
          "N" : 2363960,
          "percentile" : {
            "1.000000" : 14400,
            "5.000000" : 18560,
            "10.000000" : 18816,
            "20.000000" : 19072,
            "30.000000" : 20352,
            "40.000000" : 20352,
            "50.000000" : 20608,
            "60.000000" : 21120,
            "70.000000" : 21632,
            "80.000000" : 21888,
            "90.000000" : 22912,
            "95.000000" : 24704,
            "99.000000" : 41216,
            "99.500000" : 43776,
            "99.900000" : 55040,
            "99.950000" : 66048,
            "99.990000" : 80384
          }
        },
        "lat_ns" : {
          "min" : 5551,
          "max" : 143539,
          "mean" : 23202.878542,
          "stddev" : 3787.397119,
          "N" : 2363960
        },
        "bw_min" : 168266,
        "bw_max" : 169275,
        "bw_agg" : 100.000000,
        "bw_mean" : 168882.153846,
        "bw_dev" : 336.167579,
        "bw_samples" : 13,
        "iops_min" : 336532,
        "iops_max" : 338550,
        "iops_mean" : 337764.307692,
        "iops_stddev" : 672.444221,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.700000,
      "sys_cpu" : 83.814286,
      "ctx" : 12774,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999958,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.013790,
        "20" : 23.539004,
        "50" : 76.321145,
        "100" : 0.124791,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=143096: Wed Oct 26 04:46:25 2022
  read: IOPS=338k, BW=165MiB/s (173MB/s)(1154MiB/7001msec)
    slat (nsec): min=1122, max=22653, avg=2034.20, stdev=1011.13
    clat (nsec): min=90, max=142347, avg=21168.68, stdev=3823.82
     lat (usec): min=5, max=143, avg=23.20, stdev= 3.79
    clat percentiles (nsec):
     |  1.00th=[14400],  5.00th=[18560], 10.00th=[18816], 20.00th=[19072],
     | 30.00th=[20352], 40.00th=[20352], 50.00th=[20608], 60.00th=[21120],
     | 70.00th=[21632], 80.00th=[21888], 90.00th=[22912], 95.00th=[24704],
     | 99.00th=[41216], 99.50th=[43776], 99.90th=[55040], 99.95th=[66048],
     | 99.99th=[80384]
   bw (  KiB/s): min=168266, max=169275, per=100.00%, avg=168882.15, stdev=336.17, samples=13
   iops        : min=336532, max=338550, avg=337764.31, stdev=672.44, samples=13
  lat (nsec)   : 100=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=23.54%, 50=76.32%, 100=0.12%
  lat (usec)   : 250=0.01%
  cpu          : usr=13.70%, sys=83.81%, ctx=12774, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2363953,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=1154MiB (1210MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752398,
  "timestamp_ms" : 1666752398992,
  "time" : "Wed Oct 26 04:46:38 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1159592448,
        "io_kbytes" : 1132414,
        "bw_bytes" : 165632402,
        "bw" : 161750,
        "iops" : 323499.642908,
        "runtime" : 7001,
        "total_ios" : 2264821,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1212,
          "max" : 32892,
          "mean" : 2529.842174,
          "stddev" : 737.661078,
          "N" : 2264822
        },
        "clat_ns" : {
          "min" : 120,
          "max" : 143188,
          "mean" : 21923.724910,
          "stddev" : 3812.563652,
          "N" : 2264828,
          "percentile" : {
            "1.000000" : 14272,
            "5.000000" : 19840,
            "10.000000" : 20352,
            "20.000000" : 20864,
            "30.000000" : 21120,
            "40.000000" : 21120,
            "50.000000" : 21376,
            "60.000000" : 21376,
            "70.000000" : 21632,
            "80.000000" : 21888,
            "90.000000" : 22912,
            "95.000000" : 25984,
            "99.000000" : 42240,
            "99.500000" : 44800,
            "99.900000" : 55040,
            "99.950000" : 67072,
            "99.990000" : 81408
          }
        },
        "lat_ns" : {
          "min" : 6672,
          "max" : 144431,
          "mean" : 24453.567314,
          "stddev" : 3663.939270,
          "N" : 2264828
        },
        "bw_min" : 161462,
        "bw_max" : 162202,
        "bw_agg" : 100.000000,
        "bw_mean" : 161798.928571,
        "bw_dev" : 238.197870,
        "bw_samples" : 14,
        "iops_min" : 322924,
        "iops_max" : 324404,
        "iops_mean" : 323597.928571,
        "iops_stddev" : 476.383889,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.985714,
      "sys_cpu" : 81.714286,
      "ctx" : 12576,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999978,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.018721,
        "20" : 6.321383,
        "50" : 93.517589,
        "100" : 0.139349,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=143134: Wed Oct 26 04:46:38 2022
  read: IOPS=323k, BW=158MiB/s (166MB/s)(1106MiB/7001msec)
    slat (nsec): min=1212, max=32892, avg=2529.84, stdev=737.66
    clat (nsec): min=120, max=143188, avg=21923.72, stdev=3812.56
     lat (usec): min=6, max=144, avg=24.45, stdev= 3.66
    clat percentiles (nsec):
     |  1.00th=[14272],  5.00th=[19840], 10.00th=[20352], 20.00th=[20864],
     | 30.00th=[21120], 40.00th=[21120], 50.00th=[21376], 60.00th=[21376],
     | 70.00th=[21632], 80.00th=[21888], 90.00th=[22912], 95.00th=[25984],
     | 99.00th=[42240], 99.50th=[44800], 99.90th=[55040], 99.95th=[67072],
     | 99.99th=[81408]
   bw (  KiB/s): min=161462, max=162202, per=100.00%, avg=161798.93, stdev=238.20, samples=14
   iops        : min=322924, max=324404, avg=323597.93, stdev=476.38, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=6.32%, 50=93.52%
  lat (usec)   : 100=0.14%, 250=0.01%
  cpu          : usr=15.99%, sys=81.71%, ctx=12576, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2264821,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=1106MiB (1160MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_512_1_io_uring-reference_1.txt': No such file or directory

$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752412,
  "timestamp_ms" : 1666752412607,
  "time" : "Wed Oct 26 04:46:52 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1175407104,
        "io_kbytes" : 1147858,
        "bw_bytes" : 167891316,
        "bw" : 163956,
        "iops" : 327911.584059,
        "runtime" : 7001,
        "total_ios" : 2295709,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1202,
          "max" : 46768,
          "mean" : 2507.877895,
          "stddev" : 718.916471,
          "N" : 2295710
        },
        "clat_ns" : {
          "min" : 120,
          "max" : 143369,
          "mean" : 21623.022268,
          "stddev" : 3822.739674,
          "N" : 2295716,
          "percentile" : {
            "1.000000" : 14144,
            "5.000000" : 19840,
            "10.000000" : 20608,
            "20.000000" : 20608,
            "30.000000" : 20864,
            "40.000000" : 20864,
            "50.000000" : 21120,
            "60.000000" : 21120,
            "70.000000" : 21120,
            "80.000000" : 21376,
            "90.000000" : 22656,
            "95.000000" : 25728,
            "99.000000" : 42240,
            "99.500000" : 44288,
            "99.900000" : 56064,
            "99.950000" : 68096,
            "99.990000" : 83456
          }
        },
        "lat_ns" : {
          "min" : 6823,
          "max" : 144632,
          "mean" : 24130.900482,
          "stddev" : 3672.577138,
          "N" : 2295716
        },
        "bw_min" : 163612,
        "bw_max" : 164316,
        "bw_agg" : 100.000000,
        "bw_mean" : 164015.000000,
        "bw_dev" : 228.698856,
        "bw_samples" : 13,
        "iops_min" : 327224,
        "iops_max" : 328633,
        "iops_mean" : 328030.230769,
        "iops_stddev" : 457.340893,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.928571,
      "sys_cpu" : 81.728571,
      "ctx" : 12743,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999978,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.016378,
        "20" : 6.622529,
        "50" : 93.218435,
        "100" : 0.138563,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=143172: Wed Oct 26 04:46:52 2022
  read: IOPS=328k, BW=160MiB/s (168MB/s)(1121MiB/7001msec)
    slat (nsec): min=1202, max=46768, avg=2507.88, stdev=718.92
    clat (nsec): min=120, max=143369, avg=21623.02, stdev=3822.74
     lat (usec): min=6, max=144, avg=24.13, stdev= 3.67
    clat percentiles (nsec):
     |  1.00th=[14144],  5.00th=[19840], 10.00th=[20608], 20.00th=[20608],
     | 30.00th=[20864], 40.00th=[20864], 50.00th=[21120], 60.00th=[21120],
     | 70.00th=[21120], 80.00th=[21376], 90.00th=[22656], 95.00th=[25728],
     | 99.00th=[42240], 99.50th=[44288], 99.90th=[56064], 99.95th=[68096],
     | 99.99th=[83456]
   bw (  KiB/s): min=163612, max=164316, per=100.00%, avg=164015.00, stdev=228.70, samples=13
   iops        : min=327224, max=328633, avg=328030.23, stdev=457.34, samples=13
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=6.62%, 50=93.22%
  lat (usec)   : 100=0.14%, 250=0.01%
  cpu          : usr=15.93%, sys=81.73%, ctx=12743, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2295709,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=160MiB/s (168MB/s), 160MiB/s-160MiB/s (168MB/s-168MB/s), io=1121MiB (1175MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752426,
  "timestamp_ms" : 1666752426234,
  "time" : "Wed Oct 26 04:47:06 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1161004544,
        "io_kbytes" : 1133793,
        "bw_bytes" : 165834101,
        "bw" : 161947,
        "iops" : 323893.586630,
        "runtime" : 7001,
        "total_ios" : 2267579,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1202,
          "max" : 38503,
          "mean" : 2508.514283,
          "stddev" : 748.390671,
          "N" : 2267580
        },
        "clat_ns" : {
          "min" : 121,
          "max" : 141366,
          "mean" : 21924.472038,
          "stddev" : 3826.780599,
          "N" : 2267586,
          "percentile" : {
            "1.000000" : 14272,
            "5.000000" : 19840,
            "10.000000" : 20352,
            "20.000000" : 20864,
            "30.000000" : 21120,
            "40.000000" : 21120,
            "50.000000" : 21376,
            "60.000000" : 21376,
            "70.000000" : 21632,
            "80.000000" : 22144,
            "90.000000" : 22912,
            "95.000000" : 25728,
            "99.000000" : 42752,
            "99.500000" : 44800,
            "99.900000" : 56576,
            "99.950000" : 67072,
            "99.990000" : 81408
          }
        },
        "lat_ns" : {
          "min" : 6502,
          "max" : 142618,
          "mean" : 24432.986819,
          "stddev" : 3677.160613,
          "N" : 2267586
        },
        "bw_min" : 161484,
        "bw_max" : 162868,
        "bw_agg" : 100.000000,
        "bw_mean" : 161985.571429,
        "bw_dev" : 433.250099,
        "bw_samples" : 14,
        "iops_min" : 322966,
        "iops_max" : 325736,
        "iops_mean" : 323971.214286,
        "iops_stddev" : 866.505997,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.057143,
      "sys_cpu" : 80.628571,
      "ctx" : 12620,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999978,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.018875,
        "20" : 6.020430,
        "50" : 93.815871,
        "100" : 0.142884,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=143210: Wed Oct 26 04:47:06 2022
  read: IOPS=324k, BW=158MiB/s (166MB/s)(1107MiB/7001msec)
    slat (nsec): min=1202, max=38503, avg=2508.51, stdev=748.39
    clat (nsec): min=121, max=141366, avg=21924.47, stdev=3826.78
     lat (usec): min=6, max=142, avg=24.43, stdev= 3.68
    clat percentiles (nsec):
     |  1.00th=[14272],  5.00th=[19840], 10.00th=[20352], 20.00th=[20864],
     | 30.00th=[21120], 40.00th=[21120], 50.00th=[21376], 60.00th=[21376],
     | 70.00th=[21632], 80.00th=[22144], 90.00th=[22912], 95.00th=[25728],
     | 99.00th=[42752], 99.50th=[44800], 99.90th=[56576], 99.95th=[67072],
     | 99.99th=[81408]
   bw (  KiB/s): min=161484, max=162868, per=100.00%, avg=161985.57, stdev=433.25, samples=14
   iops        : min=322966, max=325736, avg=323971.21, stdev=866.51, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=6.02%, 50=93.82%
  lat (usec)   : 100=0.14%, 250=0.01%
  cpu          : usr=17.06%, sys=80.63%, ctx=12620, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2267579,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=1107MiB (1161MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752441,
  "timestamp_ms" : 1666752441259,
  "time" : "Wed Oct 26 04:47:21 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 2276859392,
        "io_kbytes" : 2223495,
        "bw_bytes" : 325219167,
        "bw" : 317596,
        "iops" : 635192.543922,
        "runtime" : 7001,
        "total_ios" : 4446983,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 190,
          "max" : 30808,
          "mean" : 266.992913,
          "stddev" : 64.891956,
          "N" : 4446984
        },
        "clat_ns" : {
          "min" : 441,
          "max" : 133430,
          "mean" : 11928.421873,
          "stddev" : 4431.847865,
          "N" : 4446983,
          "percentile" : {
            "1.000000" : 7008,
            "5.000000" : 7904,
            "10.000000" : 8512,
            "20.000000" : 9536,
            "30.000000" : 10048,
            "40.000000" : 10560,
            "50.000000" : 10944,
            "60.000000" : 11584,
            "70.000000" : 12224,
            "80.000000" : 13120,
            "90.000000" : 14784,
            "95.000000" : 19840,
            "99.000000" : 31104,
            "99.500000" : 34560,
            "99.900000" : 47872,
            "99.950000" : 65280,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 5991,
          "max" : 133711,
          "mean" : 12195.414785,
          "stddev" : 4429.865740,
          "N" : 4446983
        },
        "bw_min" : 317209,
        "bw_max" : 318994,
        "bw_agg" : 100.000000,
        "bw_mean" : 317838.615385,
        "bw_dev" : 507.818461,
        "bw_samples" : 13,
        "iops_min" : 634418,
        "iops_max" : 637988,
        "iops_mean" : 635677.538462,
        "iops_stddev" : 1015.420079,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 22,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 24.005084,
        "8" : 75.986191,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.992103,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 28.089853,
        "20" : 66.951931,
        "50" : 4.864332,
        "100" : 0.091613,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143259: Wed Oct 26 04:47:21 2022
  read: IOPS=635k, BW=310MiB/s (325MB/s)(2171MiB/7001msec)
    slat (nsec): min=190, max=30808, avg=266.99, stdev=64.89
    clat (nsec): min=441, max=133430, avg=11928.42, stdev=4431.85
     lat (usec): min=5, max=133, avg=12.20, stdev= 4.43
    clat percentiles (nsec):
     |  1.00th=[ 7008],  5.00th=[ 7904], 10.00th=[ 8512], 20.00th=[ 9536],
     | 30.00th=[10048], 40.00th=[10560], 50.00th=[10944], 60.00th=[11584],
     | 70.00th=[12224], 80.00th=[13120], 90.00th=[14784], 95.00th=[19840],
     | 99.00th=[31104], 99.50th=[34560], 99.90th=[47872], 99.95th=[65280],
     | 99.99th=[79360]
   bw (  KiB/s): min=317209, max=318994, per=100.00%, avg=317838.62, stdev=507.82, samples=13
   iops        : min=634418, max=637988, avg=635677.54, stdev=1015.42, samples=13
  lat (nsec)   : 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=28.09%, 20=66.95%, 50=4.86%
  lat (usec)   : 100=0.09%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=22, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=24.0%, 8=76.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4446983,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=310MiB/s (325MB/s), 310MiB/s-310MiB/s (325MB/s-325MB/s), io=2171MiB (2277MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_1.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752456,
  "timestamp_ms" : 1666752456347,
  "time" : "Wed Oct 26 04:47:36 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1845574656,
        "io_kbytes" : 1802319,
        "bw_bytes" : 263615862,
        "bw" : 257437,
        "iops" : 514873.589487,
        "runtime" : 7001,
        "total_ios" : 3604630,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 170,
          "max" : 9297,
          "mean" : 249.728569,
          "stddev" : 28.581610,
          "N" : 3604631
        },
        "clat_ns" : {
          "min" : 4809,
          "max" : 130585,
          "mean" : 14363.971523,
          "stddev" : 5215.015737,
          "N" : 3604630,
          "percentile" : {
            "1.000000" : 7456,
            "5.000000" : 8384,
            "10.000000" : 8896,
            "20.000000" : 9792,
            "30.000000" : 10560,
            "40.000000" : 11712,
            "50.000000" : 14144,
            "60.000000" : 15424,
            "70.000000" : 16512,
            "80.000000" : 18560,
            "90.000000" : 20096,
            "95.000000" : 22144,
            "99.000000" : 31616,
            "99.500000" : 35584,
            "99.900000" : 48384,
            "99.950000" : 62720,
            "99.990000" : 78336
          }
        },
        "lat_ns" : {
          "min" : 5080,
          "max" : 130805,
          "mean" : 14613.700084,
          "stddev" : 5209.006901,
          "N" : 3604630
        },
        "bw_min" : 256603,
        "bw_max" : 259025,
        "bw_agg" : 100.000000,
        "bw_mean" : 257567.500000,
        "bw_dev" : 671.520634,
        "bw_samples" : 14,
        "iops_min" : 513206,
        "iops_max" : 518050,
        "iops_mean" : 515135.142857,
        "iops_stddev" : 1342.858940,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 26,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 9.715921,
        "4" : 43.387421,
        "8" : 46.894910,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 95.677259,
        "8" : 4.322741,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 22.285450,
        "20" : 67.773863,
        "50" : 9.846531,
        "100" : 0.092104,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143310: Wed Oct 26 04:47:36 2022
  read: IOPS=515k, BW=251MiB/s (264MB/s)(1760MiB/7001msec)
    slat (nsec): min=170, max=9297, avg=249.73, stdev=28.58
    clat (usec): min=4, max=130, avg=14.36, stdev= 5.22
     lat (usec): min=5, max=130, avg=14.61, stdev= 5.21
    clat percentiles (nsec):
     |  1.00th=[ 7456],  5.00th=[ 8384], 10.00th=[ 8896], 20.00th=[ 9792],
     | 30.00th=[10560], 40.00th=[11712], 50.00th=[14144], 60.00th=[15424],
     | 70.00th=[16512], 80.00th=[18560], 90.00th=[20096], 95.00th=[22144],
     | 99.00th=[31616], 99.50th=[35584], 99.90th=[48384], 99.95th=[62720],
     | 99.99th=[78336]
   bw (  KiB/s): min=256603, max=259025, per=100.00%, avg=257567.50, stdev=671.52, samples=14
   iops        : min=513206, max=518050, avg=515135.14, stdev=1342.86, samples=14
  lat (usec)   : 10=22.29%, 20=67.77%, 50=9.85%, 100=0.09%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=26, majf=0, minf=0
  IO depths    : 1=0.1%, 2=9.7%, 4=43.4%, 8=46.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.7%, 8=4.3%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3604630,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=251MiB/s (264MB/s), 251MiB/s-251MiB/s (264MB/s-264MB/s), io=1760MiB (1846MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752471,
  "timestamp_ms" : 1666752471482,
  "time" : "Wed Oct 26 04:47:51 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 2261216256,
        "io_kbytes" : 2208219,
        "bw_bytes" : 322984753,
        "bw" : 315414,
        "iops" : 630828.595915,
        "runtime" : 7001,
        "total_ios" : 4416431,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 190,
          "max" : 24937,
          "mean" : 265.485439,
          "stddev" : 66.086546,
          "N" : 4416431
        },
        "clat_ns" : {
          "min" : 921,
          "max" : 131396,
          "mean" : 12004.528837,
          "stddev" : 4455.386441,
          "N" : 4416438,
          "percentile" : {
            "1.000000" : 6816,
            "5.000000" : 7904,
            "10.000000" : 8512,
            "20.000000" : 9536,
            "30.000000" : 10176,
            "40.000000" : 10688,
            "50.000000" : 11072,
            "60.000000" : 11712,
            "70.000000" : 12480,
            "80.000000" : 13248,
            "90.000000" : 14784,
            "95.000000" : 19840,
            "99.000000" : 31104,
            "99.500000" : 34560,
            "99.900000" : 49920,
            "99.950000" : 67072,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 5580,
          "max" : 131617,
          "mean" : 12270.014258,
          "stddev" : 4453.681752,
          "N" : 4416438
        },
        "bw_min" : 314606,
        "bw_max" : 317057,
        "bw_agg" : 100.000000,
        "bw_mean" : 315565.928571,
        "bw_dev" : 753.328967,
        "bw_samples" : 14,
        "iops_min" : 629212,
        "iops_max" : 634114,
        "iops_mean" : 631131.928571,
        "iops_stddev" : 1506.587610,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 25.115280,
        "8" : 74.875663,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.991752,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 28.039406,
        "20" : 66.984359,
        "50" : 4.876879,
        "100" : 0.097998,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143360: Wed Oct 26 04:47:51 2022
  read: IOPS=631k, BW=308MiB/s (323MB/s)(2156MiB/7001msec)
    slat (nsec): min=190, max=24937, avg=265.49, stdev=66.09
    clat (nsec): min=921, max=131396, avg=12004.53, stdev=4455.39
     lat (usec): min=5, max=131, avg=12.27, stdev= 4.45
    clat percentiles (nsec):
     |  1.00th=[ 6816],  5.00th=[ 7904], 10.00th=[ 8512], 20.00th=[ 9536],
     | 30.00th=[10176], 40.00th=[10688], 50.00th=[11072], 60.00th=[11712],
     | 70.00th=[12480], 80.00th=[13248], 90.00th=[14784], 95.00th=[19840],
     | 99.00th=[31104], 99.50th=[34560], 99.90th=[49920], 99.95th=[67072],
     | 99.99th=[79360]
   bw (  KiB/s): min=314606, max=317057, per=100.00%, avg=315565.93, stdev=753.33, samples=14
   iops        : min=629212, max=634114, avg=631131.93, stdev=1506.59, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=28.04%, 20=66.98%, 50=4.88%
  lat (usec)   : 100=0.10%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=23, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=25.1%, 8=74.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4416431,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=308MiB/s (323MB/s), 308MiB/s-308MiB/s (323MB/s-323MB/s), io=2156MiB (2261MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-reference_0.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752485,
  "timestamp_ms" : 1666752485218,
  "time" : "Wed Oct 26 04:48:05 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1124852224,
        "io_kbytes" : 1098488,
        "bw_bytes" : 160670221,
        "bw" : 156904,
        "iops" : 313807.884588,
        "runtime" : 7001,
        "total_ios" : 2196969,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1282,
          "max" : 42510,
          "mean" : 2472.162358,
          "stddev" : 910.605618,
          "N" : 2196970
        },
        "clat_ns" : {
          "min" : 80,
          "max" : 152536,
          "mean" : 22632.731950,
          "stddev" : 3858.580981,
          "N" : 2196976,
          "percentile" : {
            "1.000000" : 15552,
            "5.000000" : 20352,
            "10.000000" : 20352,
            "20.000000" : 20864,
            "30.000000" : 21632,
            "40.000000" : 21888,
            "50.000000" : 22144,
            "60.000000" : 22400,
            "70.000000" : 23168,
            "80.000000" : 23424,
            "90.000000" : 23936,
            "95.000000" : 26496,
            "99.000000" : 42752,
            "99.500000" : 45312,
            "99.900000" : 54016,
            "99.950000" : 66048,
            "99.990000" : 81408
          }
        },
        "lat_ns" : {
          "min" : 6793,
          "max" : 153869,
          "mean" : 25104.895035,
          "stddev" : 3768.630555,
          "N" : 2196976
        },
        "bw_min" : 156481,
        "bw_max" : 157772,
        "bw_agg" : 100.000000,
        "bw_mean" : 156978.923077,
        "bw_dev" : 443.567819,
        "bw_samples" : 13,
        "iops_min" : 312962,
        "iops_max" : 315544,
        "iops_mean" : 313957.923077,
        "iops_stddev" : 887.118036,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.042857,
      "sys_cpu" : 82.785714,
      "ctx" : 12432,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999954,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.022486,
        "20" : 3.535780,
        "50" : 96.305455,
        "100" : 0.129815,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 3157749,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 17441,
      "write_ticks" : 0,
      "in_queue" : 17441,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=143399: Wed Oct 26 04:48:05 2022
  read: IOPS=314k, BW=153MiB/s (161MB/s)(1073MiB/7001msec)
    slat (nsec): min=1282, max=42510, avg=2472.16, stdev=910.61
    clat (nsec): min=80, max=152536, avg=22632.73, stdev=3858.58
     lat (usec): min=6, max=153, avg=25.10, stdev= 3.77
    clat percentiles (nsec):
     |  1.00th=[15552],  5.00th=[20352], 10.00th=[20352], 20.00th=[20864],
     | 30.00th=[21632], 40.00th=[21888], 50.00th=[22144], 60.00th=[22400],
     | 70.00th=[23168], 80.00th=[23424], 90.00th=[23936], 95.00th=[26496],
     | 99.00th=[42752], 99.50th=[45312], 99.90th=[54016], 99.95th=[66048],
     | 99.99th=[81408]
   bw (  KiB/s): min=156481, max=157772, per=100.00%, avg=156978.92, stdev=443.57, samples=13
   iops        : min=312962, max=315544, avg=313957.92, stdev=887.12, samples=13
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=3.54%, 50=96.31%
  lat (usec)   : 100=0.13%, 250=0.01%
  cpu          : usr=15.04%, sys=82.79%, ctx=12432, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2196969,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=1073MiB (1125MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=3157749/0, merge=0/0, ticks=17441/0, in_queue=17441, util=99.09%

$ rm /tmp/fio-output_512_8_io_uring-reference_1.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751080,
  "timestamp_ms" : 1666751080821,
  "time" : "Wed Oct 26 04:24:40 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 284537856,
        "io_kbytes" : 277869,
        "bw_bytes" : 40642459,
        "bw" : 39689,
        "iops" : 79379.802885,
        "runtime" : 7001,
        "total_ios" : 555738,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2424,
          "max" : 32170,
          "mean" : 2650.417359,
          "stddev" : 227.123682,
          "N" : 555738
        },
        "clat_ns" : {
          "min" : 161,
          "max" : 84729,
          "mean" : 9278.076117,
          "stddev" : 1249.290259,
          "N" : 555738,
          "percentile" : {
            "1.000000" : 8896,
            "5.000000" : 8896,
            "10.000000" : 9024,
            "20.000000" : 9024,
            "30.000000" : 9024,
            "40.000000" : 9024,
            "50.000000" : 9152,
            "60.000000" : 9152,
            "70.000000" : 9152,
            "80.000000" : 9152,
            "90.000000" : 9280,
            "95.000000" : 9536,
            "99.000000" : 15296,
            "99.500000" : 15936,
            "99.900000" : 19328,
            "99.950000" : 27264,
            "99.990000" : 55552
          }
        },
        "lat_ns" : {
          "min" : 8526,
          "max" : 87364,
          "mean" : 11928.493475,
          "stddev" : 1269.367601,
          "N" : 555738
        },
        "bw_min" : 39567,
        "bw_max" : 39849,
        "bw_agg" : 100.000000,
        "bw_mean" : 39706.571429,
        "bw_dev" : 93.082027,
        "bw_samples" : 14,
        "iops_min" : 79134,
        "iops_max" : 79698,
        "iops_mean" : 79413.142857,
        "iops_stddev" : 186.164054,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.571429,
      "sys_cpu" : 49.257143,
      "ctx" : 555625,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.871188,
        "20" : 3.046580,
        "50" : 0.057581,
        "100" : 0.016555,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 797598,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 4985,
      "write_ticks" : 0,
      "in_queue" : 4985,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=139097: Wed Oct 26 04:24:40 2022
  read: IOPS=79.4k, BW=38.8MiB/s (40.6MB/s)(271MiB/7001msec)
    slat (nsec): min=2424, max=32170, avg=2650.42, stdev=227.12
    clat (nsec): min=161, max=84729, avg=9278.08, stdev=1249.29
     lat (nsec): min=8526, max=87364, avg=11928.49, stdev=1269.37
    clat percentiles (nsec):
     |  1.00th=[ 8896],  5.00th=[ 8896], 10.00th=[ 9024], 20.00th=[ 9024],
     | 30.00th=[ 9024], 40.00th=[ 9024], 50.00th=[ 9152], 60.00th=[ 9152],
     | 70.00th=[ 9152], 80.00th=[ 9152], 90.00th=[ 9280], 95.00th=[ 9536],
     | 99.00th=[15296], 99.50th=[15936], 99.90th=[19328], 99.95th=[27264],
     | 99.99th=[55552]
   bw (  KiB/s): min=39567, max=39849, per=100.00%, avg=39706.57, stdev=93.08, samples=14
   iops        : min=79134, max=79698, avg=79413.14, stdev=186.16, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 10=96.87%, 20=3.05%, 50=0.06%, 100=0.02%
  cpu          : usr=11.57%, sys=49.26%, ctx=555625, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=555738,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=38.8MiB/s (40.6MB/s), 38.8MiB/s-38.8MiB/s (40.6MB/s-40.6MB/s), io=271MiB (285MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=797598/0, merge=0/0, ticks=4985/0, in_queue=4985, util=99.09%

$ cat /tmp/fio-output_512_8_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752498,
  "timestamp_ms" : 1666752498888,
  "time" : "Wed Oct 26 04:48:18 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1118396416,
        "io_kbytes" : 1092184,
        "bw_bytes" : 159748095,
        "bw" : 156003,
        "iops" : 312006.999000,
        "runtime" : 7001,
        "total_ios" : 2184361,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1282,
          "max" : 29055,
          "mean" : 2484.140099,
          "stddev" : 984.021916,
          "N" : 2184361
        },
        "clat_ns" : {
          "min" : 70,
          "max" : 126137,
          "mean" : 22762.563038,
          "stddev" : 3843.773518,
          "N" : 2184368,
          "percentile" : {
            "1.000000" : 15936,
            "5.000000" : 20352,
            "10.000000" : 20608,
            "20.000000" : 20864,
            "30.000000" : 21632,
            "40.000000" : 21888,
            "50.000000" : 22144,
            "60.000000" : 22656,
            "70.000000" : 23424,
            "80.000000" : 23680,
            "90.000000" : 23936,
            "95.000000" : 26752,
            "99.000000" : 42752,
            "99.500000" : 45312,
            "99.900000" : 54528,
            "99.950000" : 66048,
            "99.990000" : 80384
          }
        },
        "lat_ns" : {
          "min" : 6782,
          "max" : 127519,
          "mean" : 25246.704148,
          "stddev" : 3767.691495,
          "N" : 2184368
        },
        "bw_min" : 155626,
        "bw_max" : 156619,
        "bw_agg" : 100.000000,
        "bw_mean" : 156072.692308,
        "bw_dev" : 295.570461,
        "bw_samples" : 13,
        "iops_min" : 311252,
        "iops_max" : 313238,
        "iops_mean" : 312145.692308,
        "iops_stddev" : 591.374160,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.014286,
      "sys_cpu" : 83.885714,
      "ctx" : 12410,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999954,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.028109,
        "20" : 3.435879,
        "50" : 96.395696,
        "100" : 0.135875,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 3135878,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 17469,
      "write_ticks" : 0,
      "in_queue" : 17469,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=143437: Wed Oct 26 04:48:18 2022
  read: IOPS=312k, BW=152MiB/s (160MB/s)(1067MiB/7001msec)
    slat (nsec): min=1282, max=29055, avg=2484.14, stdev=984.02
    clat (nsec): min=70, max=126137, avg=22762.56, stdev=3843.77
     lat (usec): min=6, max=127, avg=25.25, stdev= 3.77
    clat percentiles (nsec):
     |  1.00th=[15936],  5.00th=[20352], 10.00th=[20608], 20.00th=[20864],
     | 30.00th=[21632], 40.00th=[21888], 50.00th=[22144], 60.00th=[22656],
     | 70.00th=[23424], 80.00th=[23680], 90.00th=[23936], 95.00th=[26752],
     | 99.00th=[42752], 99.50th=[45312], 99.90th=[54528], 99.95th=[66048],
     | 99.99th=[80384]
   bw (  KiB/s): min=155626, max=156619, per=100.00%, avg=156072.69, stdev=295.57, samples=13
   iops        : min=311252, max=313238, avg=312145.69, stdev=591.37, samples=13
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.03%, 20=3.44%, 50=96.40%
  lat (usec)   : 100=0.14%, 250=0.01%
  cpu          : usr=14.01%, sys=83.89%, ctx=12410, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2184361,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=152MiB/s (160MB/s), 152MiB/s-152MiB/s (160MB/s-160MB/s), io=1067MiB (1118MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=3135878/0, merge=0/0, ticks=17469/0, in_queue=17469, util=99.09%

$ rm /tmp/fio-output_512_8_io_uring-reference_2.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752512,
  "timestamp_ms" : 1666752512505,
  "time" : "Wed Oct 26 04:48:32 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1115583488,
        "io_kbytes" : 1089437,
        "bw_bytes" : 159346305,
        "bw" : 155611,
        "iops" : 311222.111127,
        "runtime" : 7001,
        "total_ios" : 2178866,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1292,
          "max" : 72896,
          "mean" : 2499.882950,
          "stddev" : 1010.452035,
          "N" : 2178867
        },
        "clat_ns" : {
          "min" : 80,
          "max" : 140895,
          "mean" : 22828.623283,
          "stddev" : 3883.009481,
          "N" : 2178873,
          "percentile" : {
            "1.000000" : 15936,
            "5.000000" : 20352,
            "10.000000" : 20608,
            "20.000000" : 20864,
            "30.000000" : 21632,
            "40.000000" : 21888,
            "50.000000" : 22400,
            "60.000000" : 23168,
            "70.000000" : 23424,
            "80.000000" : 23680,
            "90.000000" : 23936,
            "95.000000" : 26752,
            "99.000000" : 42752,
            "99.500000" : 45312,
            "99.900000" : 57600,
            "99.950000" : 67072,
            "99.990000" : 81408
          }
        },
        "lat_ns" : {
          "min" : 6842,
          "max" : 142227,
          "mean" : 25328.506996,
          "stddev" : 3810.015386,
          "N" : 2178873
        },
        "bw_min" : 155507,
        "bw_max" : 155865,
        "bw_agg" : 100.000000,
        "bw_mean" : 155698.769231,
        "bw_dev" : 108.354475,
        "bw_samples" : 13,
        "iops_min" : 311014,
        "iops_max" : 311730,
        "iops_mean" : 311397.538462,
        "iops_stddev" : 216.708951,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.385714,
      "sys_cpu" : 80.500000,
      "ctx" : 12497,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999954,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.027262,
        "20" : 3.408792,
        "50" : 96.409784,
        "100" : 0.148013,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 3131246,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 17492,
      "write_ticks" : 0,
      "in_queue" : 17492,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=143476: Wed Oct 26 04:48:32 2022
  read: IOPS=311k, BW=152MiB/s (159MB/s)(1064MiB/7001msec)
    slat (nsec): min=1292, max=72896, avg=2499.88, stdev=1010.45
    clat (nsec): min=80, max=140895, avg=22828.62, stdev=3883.01
     lat (usec): min=6, max=142, avg=25.33, stdev= 3.81
    clat percentiles (nsec):
     |  1.00th=[15936],  5.00th=[20352], 10.00th=[20608], 20.00th=[20864],
     | 30.00th=[21632], 40.00th=[21888], 50.00th=[22400], 60.00th=[23168],
     | 70.00th=[23424], 80.00th=[23680], 90.00th=[23936], 95.00th=[26752],
     | 99.00th=[42752], 99.50th=[45312], 99.90th=[57600], 99.95th=[67072],
     | 99.99th=[81408]
   bw (  KiB/s): min=155507, max=155865, per=100.00%, avg=155698.77, stdev=108.35, samples=13
   iops        : min=311014, max=311730, avg=311397.54, stdev=216.71, samples=13
  lat (nsec)   : 100=0.01%, 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.03%, 20=3.41%, 50=96.41%
  lat (usec)   : 100=0.15%, 250=0.01%
  cpu          : usr=17.39%, sys=80.50%, ctx=12497, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2178866,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=1064MiB (1116MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=3131246/0, merge=0/0, ticks=17492/0, in_queue=17492, util=99.09%

$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752527,
  "timestamp_ms" : 1666752527493,
  "time" : "Wed Oct 26 04:48:47 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1236315648,
        "io_kbytes" : 1207339,
        "bw_bytes" : 176591293,
        "bw" : 172452,
        "iops" : 344903.870876,
        "runtime" : 7001,
        "total_ios" : 2414672,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 120,
          "max" : 12142,
          "mean" : 141.289363,
          "stddev" : 45.104120,
          "N" : 2414672
        },
        "clat_ns" : {
          "min" : 19116,
          "max" : 118853,
          "mean" : 21467.040025,
          "stddev" : 2785.766655,
          "N" : 2414672,
          "percentile" : {
            "1.000000" : 19584,
            "5.000000" : 19840,
            "10.000000" : 20096,
            "20.000000" : 20096,
            "30.000000" : 20352,
            "40.000000" : 20608,
            "50.000000" : 20864,
            "60.000000" : 21120,
            "70.000000" : 21376,
            "80.000000" : 21632,
            "90.000000" : 21888,
            "95.000000" : 27520,
            "99.000000" : 33536,
            "99.500000" : 37632,
            "99.900000" : 49920,
            "99.950000" : 60672,
            "99.990000" : 71168
          }
        },
        "lat_ns" : {
          "min" : 19247,
          "max" : 119083,
          "mean" : 21608.329388,
          "stddev" : 2789.818426,
          "N" : 2414672
        },
        "bw_min" : 172136,
        "bw_max" : 172888,
        "bw_agg" : 100.000000,
        "bw_mean" : 172497.071429,
        "bw_dev" : 232.881767,
        "bw_samples" : 14,
        "iops_min" : 344272,
        "iops_max" : 345776,
        "iops_mean" : 344994.142857,
        "iops_stddev" : 465.763535,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 43.014286,
      "sys_cpu" : 56.957143,
      "ctx" : 21,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 12.500000,
        "2" : 25.000000,
        "4" : 50.000000,
        "8" : 12.500000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 88.888889,
        "8" : 11.111111,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 11.204420,
        "50" : 88.695897,
        "100" : 0.098978,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=143526: Wed Oct 26 04:48:47 2022
  read: IOPS=345k, BW=168MiB/s (177MB/s)(1179MiB/7001msec)
    slat (nsec): min=120, max=12142, avg=141.29, stdev=45.10
    clat (usec): min=19, max=118, avg=21.47, stdev= 2.79
     lat (usec): min=19, max=119, avg=21.61, stdev= 2.79
    clat percentiles (nsec):
     |  1.00th=[19584],  5.00th=[19840], 10.00th=[20096], 20.00th=[20096],
     | 30.00th=[20352], 40.00th=[20608], 50.00th=[20864], 60.00th=[21120],
     | 70.00th=[21376], 80.00th=[21632], 90.00th=[21888], 95.00th=[27520],
     | 99.00th=[33536], 99.50th=[37632], 99.90th=[49920], 99.95th=[60672],
     | 99.99th=[71168]
   bw (  KiB/s): min=172136, max=172888, per=100.00%, avg=172497.07, stdev=232.88, samples=14
   iops        : min=344272, max=345776, avg=344994.14, stdev=465.76, samples=14
  lat (usec)   : 20=11.20%, 50=88.70%, 100=0.10%, 250=0.01%
  cpu          : usr=43.01%, sys=56.96%, ctx=21, majf=0, minf=0
  IO depths    : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=88.9%, 8=11.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2414672,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=168MiB/s (177MB/s), 168MiB/s-168MiB/s (177MB/s-177MB/s), io=1179MiB (1236MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752542,
  "timestamp_ms" : 1666752542587,
  "time" : "Wed Oct 26 04:49:02 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1230662144,
        "io_kbytes" : 1201818,
        "bw_bytes" : 175783765,
        "bw" : 171663,
        "iops" : 343326.667619,
        "runtime" : 7001,
        "total_ios" : 2403630,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 24797,
          "mean" : 132.649914,
          "stddev" : 44.296597,
          "N" : 2403630
        },
        "clat_ns" : {
          "min" : 16370,
          "max" : 105507,
          "mean" : 21585.856196,
          "stddev" : 2810.338268,
          "N" : 2403630,
          "percentile" : {
            "1.000000" : 19584,
            "5.000000" : 19840,
            "10.000000" : 20096,
            "20.000000" : 20352,
            "30.000000" : 20608,
            "40.000000" : 20864,
            "50.000000" : 21120,
            "60.000000" : 21120,
            "70.000000" : 21376,
            "80.000000" : 21632,
            "90.000000" : 22144,
            "95.000000" : 27008,
            "99.000000" : 35584,
            "99.500000" : 38144,
            "99.900000" : 51456,
            "99.950000" : 61184,
            "99.990000" : 71168
          }
        },
        "lat_ns" : {
          "min" : 16490,
          "max" : 105708,
          "mean" : 21718.506110,
          "stddev" : 2814.226828,
          "N" : 2403630
        },
        "bw_min" : 171328,
        "bw_max" : 172104,
        "bw_agg" : 100.000000,
        "bw_mean" : 171710.785714,
        "bw_dev" : 208.705231,
        "bw_samples" : 14,
        "iops_min" : 342656,
        "iops_max" : 344208,
        "iops_mean" : 343421.571429,
        "iops_stddev" : 417.410463,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.128571,
      "sys_cpu" : 60.842857,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 12.500010,
        "2" : 25.000021,
        "4" : 50.000000,
        "8" : 12.499969,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 88.888881,
        "8" : 11.111119,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 7.242213,
        "50" : 92.644500,
        "100" : 0.112455,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=143575: Wed Oct 26 04:49:02 2022
  read: IOPS=343k, BW=168MiB/s (176MB/s)(1174MiB/7001msec)
    slat (nsec): min=110, max=24797, avg=132.65, stdev=44.30
    clat (usec): min=16, max=105, avg=21.59, stdev= 2.81
     lat (usec): min=16, max=105, avg=21.72, stdev= 2.81
    clat percentiles (nsec):
     |  1.00th=[19584],  5.00th=[19840], 10.00th=[20096], 20.00th=[20352],
     | 30.00th=[20608], 40.00th=[20864], 50.00th=[21120], 60.00th=[21120],
     | 70.00th=[21376], 80.00th=[21632], 90.00th=[22144], 95.00th=[27008],
     | 99.00th=[35584], 99.50th=[38144], 99.90th=[51456], 99.95th=[61184],
     | 99.99th=[71168]
   bw (  KiB/s): min=171328, max=172104, per=100.00%, avg=171710.79, stdev=208.71, samples=14
   iops        : min=342656, max=344208, avg=343421.57, stdev=417.41, samples=14
  lat (usec)   : 20=7.24%, 50=92.64%, 100=0.11%, 250=0.01%
  cpu          : usr=39.13%, sys=60.84%, ctx=23, majf=0, minf=0
  IO depths    : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=88.9%, 8=11.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2403630,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s), io=1174MiB (1231MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-reference_2.txt
$ rm /tmp/fio-output_512_8_io_uring-bdev_uring_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752557,
  "timestamp_ms" : 1666752557715,
  "time" : "Wed Oct 26 04:49:17 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1238494720,
        "io_kbytes" : 1209467,
        "bw_bytes" : 176902545,
        "bw" : 172756,
        "iops" : 345511.784031,
        "runtime" : 7001,
        "total_ios" : 2418928,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 11802,
          "mean" : 126.931602,
          "stddev" : 38.723132,
          "N" : 2418928
        },
        "clat_ns" : {
          "min" : 19076,
          "max" : 118472,
          "mean" : 21508.297008,
          "stddev" : 2755.141185,
          "N" : 2418928,
          "percentile" : {
            "1.000000" : 19584,
            "5.000000" : 19840,
            "10.000000" : 20096,
            "20.000000" : 20352,
            "30.000000" : 20608,
            "40.000000" : 20864,
            "50.000000" : 20864,
            "60.000000" : 21120,
            "70.000000" : 21376,
            "80.000000" : 21632,
            "90.000000" : 22144,
            "95.000000" : 27008,
            "99.000000" : 33536,
            "99.500000" : 37632,
            "99.900000" : 50432,
            "99.950000" : 60672,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 19196,
          "max" : 118672,
          "mean" : 21635.228610,
          "stddev" : 2758.482082,
          "N" : 2418928
        },
        "bw_min" : 172512,
        "bw_max" : 173032,
        "bw_agg" : 100.000000,
        "bw_mean" : 172796.571429,
        "bw_dev" : 170.218735,
        "bw_samples" : 14,
        "iops_min" : 345024,
        "iops_max" : 346064,
        "iops_mean" : 345593.214286,
        "iops_stddev" : 340.424662,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 44.100000,
      "sys_cpu" : 55.871429,
      "ctx" : 22,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 12.500000,
        "2" : 25.000000,
        "4" : 50.000000,
        "8" : 12.500000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 88.888889,
        "8" : 11.111111,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 7.970183,
        "50" : 91.925431,
        "100" : 0.102401,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=143624: Wed Oct 26 04:49:17 2022
  read: IOPS=346k, BW=169MiB/s (177MB/s)(1181MiB/7001msec)
    slat (nsec): min=110, max=11802, avg=126.93, stdev=38.72
    clat (usec): min=19, max=118, avg=21.51, stdev= 2.76
     lat (usec): min=19, max=118, avg=21.64, stdev= 2.76
    clat percentiles (nsec):
     |  1.00th=[19584],  5.00th=[19840], 10.00th=[20096], 20.00th=[20352],
     | 30.00th=[20608], 40.00th=[20864], 50.00th=[20864], 60.00th=[21120],
     | 70.00th=[21376], 80.00th=[21632], 90.00th=[22144], 95.00th=[27008],
     | 99.00th=[33536], 99.50th=[37632], 99.90th=[50432], 99.95th=[60672],
     | 99.99th=[72192]
   bw (  KiB/s): min=172512, max=173032, per=100.00%, avg=172796.57, stdev=170.22, samples=14
   iops        : min=345024, max=346064, avg=345593.21, stdev=340.42, samples=14
  lat (usec)   : 20=7.97%, 50=91.93%, 100=0.10%, 250=0.01%
  cpu          : usr=44.10%, sys=55.87%, ctx=22, majf=0, minf=0
  IO depths    : 1=12.5%, 2=25.0%, 4=50.0%, 8=12.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=88.9%, 8=11.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2418928,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=169MiB/s (177MB/s), 169MiB/s-169MiB/s (177MB/s-177MB/s), io=1181MiB (1238MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752572,
  "timestamp_ms" : 1666752572934,
  "time" : "Wed Oct 26 04:49:32 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1087555072,
        "io_kbytes" : 1062065,
        "bw_bytes" : 155342818,
        "bw" : 151701,
        "iops" : 303402.942437,
        "runtime" : 7001,
        "total_ios" : 2124124,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1522,
          "max" : 29946,
          "mean" : 2805.505978,
          "stddev" : 1163.675313,
          "N" : 2124124
        },
        "clat_ns" : {
          "min" : 190,
          "max" : 125075,
          "mean" : 23326.719890,
          "stddev" : 2999.627077,
          "N" : 2124131,
          "percentile" : {
            "1.000000" : 16768,
            "5.000000" : 21376,
            "10.000000" : 21632,
            "20.000000" : 21888,
            "30.000000" : 22656,
            "40.000000" : 22912,
            "50.000000" : 23168,
            "60.000000" : 23168,
            "70.000000" : 23680,
            "80.000000" : 24192,
            "90.000000" : 24448,
            "95.000000" : 27264,
            "99.000000" : 34560,
            "99.500000" : 40192,
            "99.900000" : 55040,
            "99.950000" : 65280,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 6311,
          "max" : 126648,
          "mean" : 26132.225886,
          "stddev" : 2927.082286,
          "N" : 2124131
        },
        "bw_min" : 151465,
        "bw_max" : 152209,
        "bw_agg" : 100.000000,
        "bw_mean" : 151742.642857,
        "bw_dev" : 193.711366,
        "bw_samples" : 14,
        "iops_min" : 302930,
        "iops_max" : 304420,
        "iops_mean" : 303485.500000,
        "iops_stddev" : 388.155550,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 31.928571,
      "sys_cpu" : 68.042857,
      "ctx" : 26,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999976,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.030177,
        "20" : 2.877798,
        "50" : 96.939209,
        "100" : 0.136903,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143673: Wed Oct 26 04:49:32 2022
  read: IOPS=303k, BW=148MiB/s (155MB/s)(1037MiB/7001msec)
    slat (nsec): min=1522, max=29946, avg=2805.51, stdev=1163.68
    clat (nsec): min=190, max=125075, avg=23326.72, stdev=2999.63
     lat (usec): min=6, max=126, avg=26.13, stdev= 2.93
    clat percentiles (nsec):
     |  1.00th=[16768],  5.00th=[21376], 10.00th=[21632], 20.00th=[21888],
     | 30.00th=[22656], 40.00th=[22912], 50.00th=[23168], 60.00th=[23168],
     | 70.00th=[23680], 80.00th=[24192], 90.00th=[24448], 95.00th=[27264],
     | 99.00th=[34560], 99.50th=[40192], 99.90th=[55040], 99.95th=[65280],
     | 99.99th=[77312]
   bw (  KiB/s): min=151465, max=152209, per=100.00%, avg=151742.64, stdev=193.71, samples=14
   iops        : min=302930, max=304420, avg=303485.50, stdev=388.16, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.03%, 20=2.88%, 50=96.94%, 100=0.14%
  lat (usec)   : 250=0.01%
  cpu          : usr=31.93%, sys=68.04%, ctx=26, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2124124,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1037MiB (1088MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752588,
  "timestamp_ms" : 1666752588001,
  "time" : "Wed Oct 26 04:49:48 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1087149056,
        "io_kbytes" : 1061669,
        "bw_bytes" : 155284824,
        "bw" : 151645,
        "iops" : 303289.530067,
        "runtime" : 7001,
        "total_ios" : 2123330,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1523,
          "max" : 35066,
          "mean" : 2799.298077,
          "stddev" : 1164.539197,
          "N" : 2123331
        },
        "clat_ns" : {
          "min" : 200,
          "max" : 131516,
          "mean" : 23335.457212,
          "stddev" : 2973.486735,
          "N" : 2123337,
          "percentile" : {
            "1.000000" : 16768,
            "5.000000" : 21376,
            "10.000000" : 21632,
            "20.000000" : 21888,
            "30.000000" : 22656,
            "40.000000" : 22912,
            "50.000000" : 23168,
            "60.000000" : 23168,
            "70.000000" : 23680,
            "80.000000" : 24192,
            "90.000000" : 24448,
            "95.000000" : 27008,
            "99.000000" : 34560,
            "99.500000" : 39680,
            "99.900000" : 54528,
            "99.950000" : 64768,
            "99.990000" : 76288
          }
        },
        "lat_ns" : {
          "min" : 6322,
          "max" : 135254,
          "mean" : 26134.754886,
          "stddev" : 2903.757258,
          "N" : 2123337
        },
        "bw_min" : 151360,
        "bw_max" : 152173,
        "bw_agg" : 100.000000,
        "bw_mean" : 151693.142857,
        "bw_dev" : 212.960761,
        "bw_samples" : 14,
        "iops_min" : 302720,
        "iops_max" : 304346,
        "iops_mean" : 303386.500000,
        "iops_stddev" : 425.766413,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 30.957143,
      "sys_cpu" : 69.014286,
      "ctx" : 24,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999976,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.029906,
        "20" : 2.873694,
        "50" : 96.944752,
        "100" : 0.133187,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143723: Wed Oct 26 04:49:47 2022
  read: IOPS=303k, BW=148MiB/s (155MB/s)(1037MiB/7001msec)
    slat (nsec): min=1523, max=35066, avg=2799.30, stdev=1164.54
    clat (nsec): min=200, max=131516, avg=23335.46, stdev=2973.49
     lat (usec): min=6, max=135, avg=26.13, stdev= 2.90
    clat percentiles (nsec):
     |  1.00th=[16768],  5.00th=[21376], 10.00th=[21632], 20.00th=[21888],
     | 30.00th=[22656], 40.00th=[22912], 50.00th=[23168], 60.00th=[23168],
     | 70.00th=[23680], 80.00th=[24192], 90.00th=[24448], 95.00th=[27008],
     | 99.00th=[34560], 99.50th=[39680], 99.90th=[54528], 99.95th=[64768],
     | 99.99th=[76288]
   bw (  KiB/s): min=151360, max=152173, per=100.00%, avg=151693.14, stdev=212.96, samples=14
   iops        : min=302720, max=304346, avg=303386.50, stdev=425.77, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.03%, 20=2.87%, 50=96.94%, 100=0.13%
  lat (usec)   : 250=0.01%
  cpu          : usr=30.96%, sys=69.01%, ctx=24, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2123330,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1037MiB (1087MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-reference_2.txt" --cpus_allowed="1"
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752603,
  "timestamp_ms" : 1666752603063,
  "time" : "Wed Oct 26 04:50:03 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1081744896,
        "io_kbytes" : 1056391,
        "bw_bytes" : 154512911,
        "bw" : 150891,
        "iops" : 301782.031138,
        "runtime" : 7001,
        "total_ios" : 2112776,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1522,
          "max" : 27091,
          "mean" : 2821.886822,
          "stddev" : 1220.912718,
          "N" : 2112776
        },
        "clat_ns" : {
          "min" : 190,
          "max" : 120736,
          "mean" : 23444.546176,
          "stddev" : 2978.404735,
          "N" : 2112783,
          "percentile" : {
            "1.000000" : 16768,
            "5.000000" : 21376,
            "10.000000" : 21632,
            "20.000000" : 21888,
            "30.000000" : 22656,
            "40.000000" : 22912,
            "50.000000" : 23168,
            "60.000000" : 23424,
            "70.000000" : 23680,
            "80.000000" : 24192,
            "90.000000" : 24704,
            "95.000000" : 27008,
            "99.000000" : 35072,
            "99.500000" : 39680,
            "99.900000" : 51968,
            "99.950000" : 62208,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 6622,
          "max" : 122389,
          "mean" : 26266.430108,
          "stddev" : 2931.359656,
          "N" : 2112783
        },
        "bw_min" : 150430,
        "bw_max" : 151161,
        "bw_agg" : 100.000000,
        "bw_mean" : 150901.538462,
        "bw_dev" : 191.142013,
        "bw_samples" : 13,
        "iops_min" : 300860,
        "iops_max" : 302322,
        "iops_mean" : 301803.076923,
        "iops_stddev" : 382.467093,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 31.242857,
      "sys_cpu" : 68.728571,
      "ctx" : 25,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999976,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.029440,
        "20" : 2.844267,
        "50" : 96.992298,
        "100" : 0.116103,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=143773: Wed Oct 26 04:50:03 2022
  read: IOPS=302k, BW=147MiB/s (155MB/s)(1032MiB/7001msec)
    slat (nsec): min=1522, max=27091, avg=2821.89, stdev=1220.91
    clat (nsec): min=190, max=120736, avg=23444.55, stdev=2978.40
     lat (usec): min=6, max=122, avg=26.27, stdev= 2.93
    clat percentiles (nsec):
     |  1.00th=[16768],  5.00th=[21376], 10.00th=[21632], 20.00th=[21888],
     | 30.00th=[22656], 40.00th=[22912], 50.00th=[23168], 60.00th=[23424],
     | 70.00th=[23680], 80.00th=[24192], 90.00th=[24704], 95.00th=[27008],
     | 99.00th=[35072], 99.50th=[39680], 99.90th=[51968], 99.95th=[62208],
     | 99.99th=[74240]
   bw (  KiB/s): min=150430, max=151161, per=100.00%, avg=150901.54, stdev=191.14, samples=13
   iops        : min=300860, max=302322, avg=301803.08, stdev=382.47, samples=13
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.03%, 20=2.84%, 50=96.99%, 100=0.12%
  lat (usec)   : 250=0.01%
  cpu          : usr=31.24%, sys=68.73%, ctx=25, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2112776,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=147MiB/s (155MB/s), 147MiB/s-147MiB/s (155MB/s-155MB/s), io=1032MiB (1082MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752616,
  "timestamp_ms" : 1666752616765,
  "time" : "Wed Oct 26 04:50:16 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1090587136,
        "io_kbytes" : 1065026,
        "bw_bytes" : 155775908,
        "bw" : 152124,
        "iops" : 304248.678760,
        "runtime" : 7001,
        "total_ios" : 2130045,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1382,
          "max" : 75892,
          "mean" : 2807.293279,
          "stddev" : 1237.975793,
          "N" : 2130046
        },
        "clat_ns" : {
          "min" : 110,
          "max" : 140474,
          "mean" : 23243.132478,
          "stddev" : 4091.818656,
          "N" : 2130052,
          "percentile" : {
            "1.000000" : 15424,
            "5.000000" : 19328,
            "10.000000" : 19584,
            "20.000000" : 21120,
            "30.000000" : 21632,
            "40.000000" : 22656,
            "50.000000" : 22912,
            "60.000000" : 23168,
            "70.000000" : 23936,
            "80.000000" : 24704,
            "90.000000" : 25984,
            "95.000000" : 27264,
            "99.000000" : 43264,
            "99.500000" : 45824,
            "99.900000" : 56576,
            "99.950000" : 66048,
            "99.990000" : 81408
          }
        },
        "lat_ns" : {
          "min" : 6923,
          "max" : 141916,
          "mean" : 26050.426103,
          "stddev" : 4239.486761,
          "N" : 2130052
        },
        "bw_min" : 151886,
        "bw_max" : 152476,
        "bw_agg" : 100.000000,
        "bw_mean" : 152167.357143,
        "bw_dev" : 193.514304,
        "bw_samples" : 14,
        "iops_min" : 303772,
        "iops_max" : 304952,
        "iops_mean" : 304334.714286,
        "iops_stddev" : 386.864800,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.885714,
      "sys_cpu" : 80.342857,
      "ctx" : 11485,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999977,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.044882,
        "20" : 15.144750,
        "50" : 84.647883,
        "100" : 0.147649,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=143812: Wed Oct 26 04:50:16 2022
  read: IOPS=304k, BW=149MiB/s (156MB/s)(1040MiB/7001msec)
    slat (nsec): min=1382, max=75892, avg=2807.29, stdev=1237.98
    clat (nsec): min=110, max=140474, avg=23243.13, stdev=4091.82
     lat (usec): min=6, max=141, avg=26.05, stdev= 4.24
    clat percentiles (nsec):
     |  1.00th=[15424],  5.00th=[19328], 10.00th=[19584], 20.00th=[21120],
     | 30.00th=[21632], 40.00th=[22656], 50.00th=[22912], 60.00th=[23168],
     | 70.00th=[23936], 80.00th=[24704], 90.00th=[25984], 95.00th=[27264],
     | 99.00th=[43264], 99.50th=[45824], 99.90th=[56576], 99.95th=[66048],
     | 99.99th=[81408]
   bw (  KiB/s): min=151886, max=152476, per=100.00%, avg=152167.36, stdev=193.51, samples=14
   iops        : min=303772, max=304952, avg=304334.71, stdev=386.86, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.04%, 20=15.14%, 50=84.65%
  lat (usec)   : 100=0.15%, 250=0.01%
  cpu          : usr=17.89%, sys=80.34%, ctx=11485, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2130045,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=1040MiB (1091MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752630,
  "timestamp_ms" : 1666752630403,
  "time" : "Wed Oct 26 04:50:30 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1091710976,
        "io_kbytes" : 1066124,
        "bw_bytes" : 155936434,
        "bw" : 152281,
        "iops" : 304562.205399,
        "runtime" : 7001,
        "total_ios" : 2132240,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1372,
          "max" : 65753,
          "mean" : 2806.080467,
          "stddev" : 1233.481209,
          "N" : 2132241
        },
        "clat_ns" : {
          "min" : 110,
          "max" : 142047,
          "mean" : 23217.680802,
          "stddev" : 4093.965853,
          "N" : 2132247,
          "percentile" : {
            "1.000000" : 15424,
            "5.000000" : 19328,
            "10.000000" : 19584,
            "20.000000" : 21120,
            "30.000000" : 21632,
            "40.000000" : 22656,
            "50.000000" : 22912,
            "60.000000" : 23168,
            "70.000000" : 23936,
            "80.000000" : 24704,
            "90.000000" : 25984,
            "95.000000" : 27264,
            "99.000000" : 43264,
            "99.500000" : 45824,
            "99.900000" : 57600,
            "99.950000" : 67072,
            "99.990000" : 81408
          }
        },
        "lat_ns" : {
          "min" : 6883,
          "max" : 143439,
          "mean" : 26023.761436,
          "stddev" : 4231.145211,
          "N" : 2132247
        },
        "bw_min" : 152213,
        "bw_max" : 152540,
        "bw_agg" : 100.000000,
        "bw_mean" : 152322.357143,
        "bw_dev" : 94.171372,
        "bw_samples" : 14,
        "iops_min" : 304426,
        "iops_max" : 305080,
        "iops_mean" : 304644.857143,
        "iops_stddev" : 188.408254,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.142857,
      "sys_cpu" : 79.042857,
      "ctx" : 11717,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999977,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.046196,
        "20" : 14.669643,
        "50" : 85.119499,
        "100" : 0.149655,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=143851: Wed Oct 26 04:50:30 2022
  read: IOPS=305k, BW=149MiB/s (156MB/s)(1041MiB/7001msec)
    slat (nsec): min=1372, max=65753, avg=2806.08, stdev=1233.48
    clat (nsec): min=110, max=142047, avg=23217.68, stdev=4093.97
     lat (usec): min=6, max=143, avg=26.02, stdev= 4.23
    clat percentiles (nsec):
     |  1.00th=[15424],  5.00th=[19328], 10.00th=[19584], 20.00th=[21120],
     | 30.00th=[21632], 40.00th=[22656], 50.00th=[22912], 60.00th=[23168],
     | 70.00th=[23936], 80.00th=[24704], 90.00th=[25984], 95.00th=[27264],
     | 99.00th=[43264], 99.50th=[45824], 99.90th=[57600], 99.95th=[67072],
     | 99.99th=[81408]
   bw (  KiB/s): min=152213, max=152540, per=100.00%, avg=152322.36, stdev=94.17, samples=14
   iops        : min=304426, max=305080, avg=304644.86, stdev=188.41, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.05%, 20=14.67%, 50=85.12%
  lat (usec)   : 100=0.15%, 250=0.01%
  cpu          : usr=19.14%, sys=79.04%, ctx=11717, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2132240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=1041MiB (1092MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_io_uring-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751094,
  "timestamp_ms" : 1666751094444,
  "time" : "Wed Oct 26 04:24:54 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 286309376,
        "io_kbytes" : 279599,
        "bw_bytes" : 40895497,
        "bw" : 39937,
        "iops" : 79874.017997,
        "runtime" : 7001,
        "total_ios" : 559198,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2494,
          "max" : 45485,
          "mean" : 2642.445052,
          "stddev" : 228.167226,
          "N" : 559198
        },
        "clat_ns" : {
          "min" : 190,
          "max" : 81473,
          "mean" : 9223.417267,
          "stddev" : 1276.987473,
          "N" : 559198,
          "percentile" : {
            "1.000000" : 8768,
            "5.000000" : 8896,
            "10.000000" : 8896,
            "20.000000" : 9024,
            "30.000000" : 9024,
            "40.000000" : 9024,
            "50.000000" : 9024,
            "60.000000" : 9024,
            "70.000000" : 9152,
            "80.000000" : 9152,
            "90.000000" : 9152,
            "95.000000" : 9408,
            "99.000000" : 15424,
            "99.500000" : 16192,
            "99.900000" : 19584,
            "99.950000" : 23168,
            "99.990000" : 56576
          }
        },
        "lat_ns" : {
          "min" : 8957,
          "max" : 84108,
          "mean" : 11865.862319,
          "stddev" : 1296.542780,
          "N" : 559198
        },
        "bw_min" : 39762,
        "bw_max" : 40171,
        "bw_agg" : 100.000000,
        "bw_mean" : 39954.785714,
        "bw_dev" : 110.705554,
        "bw_samples" : 14,
        "iops_min" : 79524,
        "iops_max" : 80342,
        "iops_mean" : 79909.642857,
        "iops_stddev" : 221.379454,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.328571,
      "sys_cpu" : 46.328571,
      "ctx" : 559054,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 97.150741,
        "20" : 2.758057,
        "50" : 0.062590,
        "100" : 0.018062,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 801785,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5027,
      "write_ticks" : 0,
      "in_queue" : 5027,
      "util" : 99.098564
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=139136: Wed Oct 26 04:24:54 2022
  read: IOPS=79.9k, BW=39.0MiB/s (40.9MB/s)(273MiB/7001msec)
    slat (nsec): min=2494, max=45485, avg=2642.45, stdev=228.17
    clat (nsec): min=190, max=81473, avg=9223.42, stdev=1276.99
     lat (nsec): min=8957, max=84108, avg=11865.86, stdev=1296.54
    clat percentiles (nsec):
     |  1.00th=[ 8768],  5.00th=[ 8896], 10.00th=[ 8896], 20.00th=[ 9024],
     | 30.00th=[ 9024], 40.00th=[ 9024], 50.00th=[ 9024], 60.00th=[ 9024],
     | 70.00th=[ 9152], 80.00th=[ 9152], 90.00th=[ 9152], 95.00th=[ 9408],
     | 99.00th=[15424], 99.50th=[16192], 99.90th=[19584], 99.95th=[23168],
     | 99.99th=[56576]
   bw (  KiB/s): min=39762, max=40171, per=100.00%, avg=39954.79, stdev=110.71, samples=14
   iops        : min=79524, max=80342, avg=79909.64, stdev=221.38, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 10=97.15%, 20=2.76%, 50=0.06%, 100=0.02%
  cpu          : usr=14.33%, sys=46.33%, ctx=559054, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=559198,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=39.0MiB/s (40.9MB/s), 39.0MiB/s-39.0MiB/s (40.9MB/s-40.9MB/s), io=273MiB (286MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=801785/0, merge=0/0, ticks=5027/0, in_queue=5027, util=99.10%

$ cat /tmp/fio-output_512_8_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752644,
  "timestamp_ms" : 1666752644045,
  "time" : "Wed Oct 26 04:50:44 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1097063424,
        "io_kbytes" : 1071351,
        "bw_bytes" : 156700960,
        "bw" : 153028,
        "iops" : 306055.420654,
        "runtime" : 7001,
        "total_ios" : 2142694,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1362,
          "max" : 43081,
          "mean" : 2770.524205,
          "stddev" : 1257.270481,
          "N" : 2142695
        },
        "clat_ns" : {
          "min" : 110,
          "max" : 142568,
          "mean" : 23118.608042,
          "stddev" : 3988.582641,
          "N" : 2142701,
          "percentile" : {
            "1.000000" : 15424,
            "5.000000" : 19328,
            "10.000000" : 20864,
            "20.000000" : 21120,
            "30.000000" : 21376,
            "40.000000" : 22400,
            "50.000000" : 22656,
            "60.000000" : 22912,
            "70.000000" : 23680,
            "80.000000" : 24192,
            "90.000000" : 25472,
            "95.000000" : 27776,
            "99.000000" : 42752,
            "99.500000" : 45824,
            "99.900000" : 58624,
            "99.950000" : 68096,
            "99.990000" : 82432
          }
        },
        "lat_ns" : {
          "min" : 6923,
          "max" : 143950,
          "mean" : 25889.131866,
          "stddev" : 4007.357932,
          "N" : 2142701
        },
        "bw_min" : 152809,
        "bw_max" : 153283,
        "bw_agg" : 100.000000,
        "bw_mean" : 153068.142857,
        "bw_dev" : 168.665740,
        "bw_samples" : 14,
        "iops_min" : 305618,
        "iops_max" : 306566,
        "iops_mean" : 306136.214286,
        "iops_stddev" : 337.609738,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.928571,
      "sys_cpu" : 78.200000,
      "ctx" : 12004,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999977,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.040043,
        "20" : 8.114318,
        "50" : 91.677533,
        "100" : 0.156905,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=143892: Wed Oct 26 04:50:44 2022
  read: IOPS=306k, BW=149MiB/s (157MB/s)(1046MiB/7001msec)
    slat (nsec): min=1362, max=43081, avg=2770.52, stdev=1257.27
    clat (nsec): min=110, max=142568, avg=23118.61, stdev=3988.58
     lat (usec): min=6, max=143, avg=25.89, stdev= 4.01
    clat percentiles (nsec):
     |  1.00th=[15424],  5.00th=[19328], 10.00th=[20864], 20.00th=[21120],
     | 30.00th=[21376], 40.00th=[22400], 50.00th=[22656], 60.00th=[22912],
     | 70.00th=[23680], 80.00th=[24192], 90.00th=[25472], 95.00th=[27776],
     | 99.00th=[42752], 99.50th=[45824], 99.90th=[58624], 99.95th=[68096],
     | 99.99th=[82432]
   bw (  KiB/s): min=152809, max=153283, per=100.00%, avg=153068.14, stdev=168.67, samples=14
   iops        : min=305618, max=306566, avg=306136.21, stdev=337.61, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.04%, 20=8.11%, 50=91.68%
  lat (usec)   : 100=0.16%, 250=0.01%
  cpu          : usr=19.93%, sys=78.20%, ctx=12004, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2142694,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=149MiB/s (157MB/s), 149MiB/s-149MiB/s (157MB/s-157MB/s), io=1046MiB (1097MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-reference_0.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752657,
  "timestamp_ms" : 1666752657713,
  "time" : "Wed Oct 26 04:50:57 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1046272512,
        "io_kbytes" : 1021750,
        "bw_bytes" : 149446152,
        "bw" : 145943,
        "iops" : 291885.873447,
        "runtime" : 7001,
        "total_ios" : 2043493,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1402,
          "max" : 113964,
          "mean" : 2707.785157,
          "stddev" : 1337.154939,
          "N" : 2043494
        },
        "clat_ns" : {
          "min" : 2595,
          "max" : 138590,
          "mean" : 24451.180669,
          "stddev" : 3177.840339,
          "N" : 2043500,
          "percentile" : {
            "1.000000" : 17536,
            "5.000000" : 22400,
            "10.000000" : 22656,
            "20.000000" : 22656,
            "30.000000" : 22912,
            "40.000000" : 23680,
            "50.000000" : 24192,
            "60.000000" : 24448,
            "70.000000" : 25216,
            "80.000000" : 25472,
            "90.000000" : 25728,
            "95.000000" : 29312,
            "99.000000" : 37632,
            "99.500000" : 42240,
            "99.900000" : 54528,
            "99.950000" : 67072,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 10450,
          "max" : 143930,
          "mean" : 27158.965725,
          "stddev" : 3022.018116,
          "N" : 2043500
        },
        "bw_min" : 145819,
        "bw_max" : 146239,
        "bw_agg" : 100.000000,
        "bw_mean" : 146001.692308,
        "bw_dev" : 141.640028,
        "bw_samples" : 13,
        "iops_min" : 291638,
        "iops_max" : 292478,
        "iops_mean" : 292003.461538,
        "iops_stddev" : 283.176216,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 35.971429,
      "sys_cpu" : 62.771429,
      "ctx" : 10182,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999951,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 2.033870,
        "50" : 97.828033,
        "100" : 0.127282,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2858731,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 15931,
      "write_ticks" : 0,
      "in_queue" : 15931,
      "util" : 99.035239
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=143932: Wed Oct 26 04:50:57 2022
  read: IOPS=292k, BW=143MiB/s (149MB/s)(998MiB/7001msec)
    slat (nsec): min=1402, max=113964, avg=2707.79, stdev=1337.15
    clat (usec): min=2, max=138, avg=24.45, stdev= 3.18
     lat (usec): min=10, max=143, avg=27.16, stdev= 3.02
    clat percentiles (nsec):
     |  1.00th=[17536],  5.00th=[22400], 10.00th=[22656], 20.00th=[22656],
     | 30.00th=[22912], 40.00th=[23680], 50.00th=[24192], 60.00th=[24448],
     | 70.00th=[25216], 80.00th=[25472], 90.00th=[25728], 95.00th=[29312],
     | 99.00th=[37632], 99.50th=[42240], 99.90th=[54528], 99.95th=[67072],
     | 99.99th=[79360]
   bw (  KiB/s): min=145819, max=146239, per=100.00%, avg=146001.69, stdev=141.64, samples=13
   iops        : min=291638, max=292478, avg=292003.46, stdev=283.18, samples=13
  lat (usec)   : 4=0.01%, 10=0.01%, 20=2.03%, 50=97.83%, 100=0.13%
  lat (usec)   : 250=0.01%
  cpu          : usr=35.97%, sys=62.77%, ctx=10182, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2043493,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=143MiB/s (149MB/s), 143MiB/s-143MiB/s (149MB/s-149MB/s), io=998MiB (1046MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2858731/0, merge=0/0, ticks=15931/0, in_queue=15931, util=99.04%

$ rm /tmp/fio-output_512_8_libaio-reference_1.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752671,
  "timestamp_ms" : 1666752671349,
  "time" : "Wed Oct 26 04:51:11 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1046752768,
        "io_kbytes" : 1022219,
        "bw_bytes" : 149514750,
        "bw" : 146010,
        "iops" : 292019.854307,
        "runtime" : 7001,
        "total_ios" : 2044431,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1403,
          "max" : 123251,
          "mean" : 2715.633511,
          "stddev" : 1330.313775,
          "N" : 2044432
        },
        "clat_ns" : {
          "min" : 2605,
          "max" : 145643,
          "mean" : 24432.144750,
          "stddev" : 3169.438036,
          "N" : 2044438,
          "percentile" : {
            "1.000000" : 17536,
            "5.000000" : 22400,
            "10.000000" : 22656,
            "20.000000" : 22656,
            "30.000000" : 22912,
            "40.000000" : 23680,
            "50.000000" : 23936,
            "60.000000" : 24448,
            "70.000000" : 24960,
            "80.000000" : 25472,
            "90.000000" : 25728,
            "95.000000" : 29312,
            "99.000000" : 37632,
            "99.500000" : 41728,
            "99.900000" : 52992,
            "99.950000" : 65280,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 10781,
          "max" : 148709,
          "mean" : 27147.777613,
          "stddev" : 3031.587372,
          "N" : 2044438
        },
        "bw_min" : 145798,
        "bw_max" : 146766,
        "bw_agg" : 100.000000,
        "bw_mean" : 146077.000000,
        "bw_dev" : 239.693394,
        "bw_samples" : 14,
        "iops_min" : 291596,
        "iops_max" : 293532,
        "iops_mean" : 292154.142857,
        "iops_stddev" : 479.286466,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 38.614286,
      "sys_cpu" : 60.157143,
      "ctx" : 10093,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999951,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 2.060720,
        "50" : 97.807507,
        "100" : 0.119887,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2933310,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 16349,
      "write_ticks" : 0,
      "in_queue" : 16349,
      "util" : 99.098564
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=143970: Wed Oct 26 04:51:11 2022
  read: IOPS=292k, BW=143MiB/s (150MB/s)(998MiB/7001msec)
    slat (nsec): min=1403, max=123251, avg=2715.63, stdev=1330.31
    clat (usec): min=2, max=145, avg=24.43, stdev= 3.17
     lat (usec): min=10, max=148, avg=27.15, stdev= 3.03
    clat percentiles (nsec):
     |  1.00th=[17536],  5.00th=[22400], 10.00th=[22656], 20.00th=[22656],
     | 30.00th=[22912], 40.00th=[23680], 50.00th=[23936], 60.00th=[24448],
     | 70.00th=[24960], 80.00th=[25472], 90.00th=[25728], 95.00th=[29312],
     | 99.00th=[37632], 99.50th=[41728], 99.90th=[52992], 99.95th=[65280],
     | 99.99th=[79360]
   bw (  KiB/s): min=145798, max=146766, per=100.00%, avg=146077.00, stdev=239.69, samples=14
   iops        : min=291596, max=293532, avg=292154.14, stdev=479.29, samples=14
  lat (usec)   : 4=0.01%, 10=0.01%, 20=2.06%, 50=97.81%, 100=0.12%
  lat (usec)   : 250=0.01%
  cpu          : usr=38.61%, sys=60.16%, ctx=10093, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2044431,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=998MiB (1047MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2933310/0, merge=0/0, ticks=16349/0, in_queue=16349, util=99.10%

$ rm /tmp/fio-output_512_8_libaio-reference_2.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752685,
  "timestamp_ms" : 1666752685000,
  "time" : "Wed Oct 26 04:51:25 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1043872256,
        "io_kbytes" : 1019406,
        "bw_bytes" : 149103307,
        "bw" : 145608,
        "iops" : 291216.254821,
        "runtime" : 7001,
        "total_ios" : 2038805,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1432,
          "max" : 27141,
          "mean" : 2732.441553,
          "stddev" : 1335.007083,
          "N" : 2038806
        },
        "clat_ns" : {
          "min" : 4679,
          "max" : 159368,
          "mean" : 24492.867077,
          "stddev" : 3138.428213,
          "N" : 2038812,
          "percentile" : {
            "1.000000" : 17536,
            "5.000000" : 22400,
            "10.000000" : 22656,
            "20.000000" : 22912,
            "30.000000" : 22912,
            "40.000000" : 23680,
            "50.000000" : 24192,
            "60.000000" : 24960,
            "70.000000" : 25216,
            "80.000000" : 25472,
            "90.000000" : 25728,
            "95.000000" : 29312,
            "99.000000" : 37632,
            "99.500000" : 41728,
            "99.900000" : 52480,
            "99.950000" : 65280,
            "99.990000" : 78336
          }
        },
        "lat_ns" : {
          "min" : 10911,
          "max" : 160841,
          "mean" : 27225.311592,
          "stddev" : 2968.109528,
          "N" : 2038812
        },
        "bw_min" : 145239,
        "bw_max" : 145957,
        "bw_agg" : 100.000000,
        "bw_mean" : 145648.230769,
        "bw_dev" : 185.987972,
        "bw_samples" : 13,
        "iops_min" : 290478,
        "iops_max" : 291914,
        "iops_mean" : 291296.538462,
        "iops_stddev" : 372.038218,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 37.700000,
      "sys_cpu" : 61.085714,
      "ctx" : 10154,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999951,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 1.971253,
        "50" : 97.900584,
        "100" : 0.117912,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2855695,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 15886,
      "write_ticks" : 0,
      "in_queue" : 15886,
      "util" : 99.045298
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=144008: Wed Oct 26 04:51:24 2022
  read: IOPS=291k, BW=142MiB/s (149MB/s)(996MiB/7001msec)
    slat (nsec): min=1432, max=27141, avg=2732.44, stdev=1335.01
    clat (usec): min=4, max=159, avg=24.49, stdev= 3.14
     lat (usec): min=10, max=160, avg=27.23, stdev= 2.97
    clat percentiles (nsec):
     |  1.00th=[17536],  5.00th=[22400], 10.00th=[22656], 20.00th=[22912],
     | 30.00th=[22912], 40.00th=[23680], 50.00th=[24192], 60.00th=[24960],
     | 70.00th=[25216], 80.00th=[25472], 90.00th=[25728], 95.00th=[29312],
     | 99.00th=[37632], 99.50th=[41728], 99.90th=[52480], 99.95th=[65280],
     | 99.99th=[78336]
   bw (  KiB/s): min=145239, max=145957, per=100.00%, avg=145648.23, stdev=185.99, samples=13
   iops        : min=290478, max=291914, avg=291296.54, stdev=372.04, samples=13
  lat (usec)   : 10=0.01%, 20=1.97%, 50=97.90%, 100=0.12%, 250=0.01%
  cpu          : usr=37.70%, sys=61.09%, ctx=10154, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2038805,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=996MiB (1044MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2855695/0, merge=0/0, ticks=15886/0, in_queue=15886, util=99.05%

$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_0.txt
$ rm /tmp/fio-output_512_8_libaio-bdev_aio_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752700,
  "timestamp_ms" : 1666752700018,
  "time" : "Wed Oct 26 04:51:40 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1086418944,
        "io_kbytes" : 1060956,
        "bw_bytes" : 155180537,
        "bw" : 151543,
        "iops" : 303086.559063,
        "runtime" : 7001,
        "total_ios" : 2121909,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1593,
          "max" : 35116,
          "mean" : 2769.186245,
          "stddev" : 1002.279645,
          "N" : 2121910
        },
        "clat_ns" : {
          "min" : 330,
          "max" : 188353,
          "mean" : 14078.489138,
          "stddev" : 6683.083761,
          "N" : 2121912,
          "percentile" : {
            "1.000000" : 3408,
            "5.000000" : 3536,
            "10.000000" : 4832,
            "20.000000" : 6816,
            "30.000000" : 9664,
            "40.000000" : 12096,
            "50.000000" : 14272,
            "60.000000" : 16320,
            "70.000000" : 18816,
            "80.000000" : 20608,
            "90.000000" : 22912,
            "95.000000" : 23424,
            "99.000000" : 27264,
            "99.500000" : 29824,
            "99.900000" : 42752,
            "99.950000" : 57088,
            "99.990000" : 70144
          }
        },
        "lat_ns" : {
          "min" : 6182,
          "max" : 191369,
          "mean" : 16847.674926,
          "stddev" : 6639.843683,
          "N" : 2121912
        },
        "bw_min" : 150945,
        "bw_max" : 152439,
        "bw_agg" : 100.000000,
        "bw_mean" : 151590.285714,
        "bw_dev" : 485.280320,
        "bw_samples" : 14,
        "iops_min" : 301890,
        "iops_max" : 304878,
        "iops_mean" : 303180.571429,
        "iops_stddev" : 970.850675,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 40.557143,
      "sys_cpu" : 59.414286,
      "ctx" : 26,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.471274,
        "2" : 26.285576,
        "4" : 58.295997,
        "8" : 14.947154,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 87.443970,
        "8" : 12.556030,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.033743,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 6.563524,
        "10" : 29.667955,
        "20" : 42.265856,
        "50" : 21.394131,
        "100" : 0.072718,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=144056: Wed Oct 26 04:51:40 2022
  read: IOPS=303k, BW=148MiB/s (155MB/s)(1036MiB/7001msec)
    slat (nsec): min=1593, max=35116, avg=2769.19, stdev=1002.28
    clat (nsec): min=330, max=188353, avg=14078.49, stdev=6683.08
     lat (usec): min=6, max=191, avg=16.85, stdev= 6.64
    clat percentiles (nsec):
     |  1.00th=[ 3408],  5.00th=[ 3536], 10.00th=[ 4832], 20.00th=[ 6816],
     | 30.00th=[ 9664], 40.00th=[12096], 50.00th=[14272], 60.00th=[16320],
     | 70.00th=[18816], 80.00th=[20608], 90.00th=[22912], 95.00th=[23424],
     | 99.00th=[27264], 99.50th=[29824], 99.90th=[42752], 99.95th=[57088],
     | 99.99th=[70144]
   bw (  KiB/s): min=150945, max=152439, per=100.00%, avg=151590.29, stdev=485.28, samples=14
   iops        : min=301890, max=304878, avg=303180.57, stdev=970.85, samples=14
  lat (nsec)   : 500=0.03%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=6.56%, 10=29.67%, 20=42.27%, 50=21.39%
  lat (usec)   : 100=0.07%, 250=0.01%
  cpu          : usr=40.56%, sys=59.41%, ctx=26, majf=0, minf=0
  IO depths    : 1=0.5%, 2=26.3%, 4=58.3%, 8=14.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=87.4%, 8=12.6%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2121909,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=1036MiB (1086MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-bdev_aio_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752715,
  "timestamp_ms" : 1666752715116,
  "time" : "Wed Oct 26 04:51:55 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1076485120,
        "io_kbytes" : 1051255,
        "bw_bytes" : 153761622,
        "bw" : 150157,
        "iops" : 300314.669333,
        "runtime" : 7001,
        "total_ios" : 2102503,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1593,
          "max" : 36158,
          "mean" : 2832.892318,
          "stddev" : 1021.358270,
          "N" : 2102503
        },
        "clat_ns" : {
          "min" : 350,
          "max" : 128501,
          "mean" : 14078.870878,
          "stddev" : 6722.192899,
          "N" : 2102503,
          "percentile" : {
            "1.000000" : 3440,
            "5.000000" : 3568,
            "10.000000" : 4832,
            "20.000000" : 6816,
            "30.000000" : 9664,
            "40.000000" : 11712,
            "50.000000" : 14144,
            "60.000000" : 16320,
            "70.000000" : 18304,
            "80.000000" : 19840,
            "90.000000" : 23168,
            "95.000000" : 23680,
            "99.000000" : 27008,
            "99.500000" : 29824,
            "99.900000" : 44800,
            "99.950000" : 57600,
            "99.990000" : 69120
          }
        },
        "lat_ns" : {
          "min" : 6161,
          "max" : 130174,
          "mean" : 16911.763196,
          "stddev" : 6736.191451,
          "N" : 2102503
        },
        "bw_min" : 149226,
        "bw_max" : 150989,
        "bw_agg" : 100.000000,
        "bw_mean" : 150194.785714,
        "bw_dev" : 508.754464,
        "bw_samples" : 14,
        "iops_min" : 298452,
        "iops_max" : 301979,
        "iops_mean" : 300389.642857,
        "iops_stddev" : 1017.629040,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 40.442857,
      "sys_cpu" : 59.528571,
      "ctx" : 29,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.466064,
        "2" : 26.752542,
        "4" : 57.926671,
        "8" : 14.854723,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 87.516984,
        "8" : 12.483016,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.042806,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 6.502393,
        "10" : 29.740600,
        "20" : 43.938059,
        "50" : 19.695097,
        "100" : 0.078906,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=144105: Wed Oct 26 04:51:55 2022
  read: IOPS=300k, BW=147MiB/s (154MB/s)(1027MiB/7001msec)
    slat (nsec): min=1593, max=36158, avg=2832.89, stdev=1021.36
    clat (nsec): min=350, max=128501, avg=14078.87, stdev=6722.19
     lat (usec): min=6, max=130, avg=16.91, stdev= 6.74
    clat percentiles (nsec):
     |  1.00th=[ 3440],  5.00th=[ 3568], 10.00th=[ 4832], 20.00th=[ 6816],
     | 30.00th=[ 9664], 40.00th=[11712], 50.00th=[14144], 60.00th=[16320],
     | 70.00th=[18304], 80.00th=[19840], 90.00th=[23168], 95.00th=[23680],
     | 99.00th=[27008], 99.50th=[29824], 99.90th=[44800], 99.95th=[57600],
     | 99.99th=[69120]
   bw (  KiB/s): min=149226, max=150989, per=100.00%, avg=150194.79, stdev=508.75, samples=14
   iops        : min=298452, max=301979, avg=300389.64, stdev=1017.63, samples=14
  lat (nsec)   : 500=0.04%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=6.50%, 10=29.74%, 20=43.94%, 50=19.70%
  lat (usec)   : 100=0.08%, 250=0.01%
  cpu          : usr=40.44%, sys=59.53%, ctx=29, majf=0, minf=0
  IO depths    : 1=0.5%, 2=26.8%, 4=57.9%, 8=14.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=87.5%, 8=12.5%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2102503,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=1027MiB (1076MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-bdev_aio_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752730,
  "timestamp_ms" : 1666752730123,
  "time" : "Wed Oct 26 04:52:10 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1091307008,
        "io_kbytes" : 1065729,
        "bw_bytes" : 155878732,
        "bw" : 152225,
        "iops" : 304450.078560,
        "runtime" : 7001,
        "total_ios" : 2131455,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1572,
          "max" : 80341,
          "mean" : 2723.375660,
          "stddev" : 926.510920,
          "N" : 2131456
        },
        "clat_ns" : {
          "min" : 320,
          "max" : 145703,
          "mean" : 14031.642536,
          "stddev" : 6669.051166,
          "N" : 2131459,
          "percentile" : {
            "1.000000" : 3376,
            "5.000000" : 3504,
            "10.000000" : 4832,
            "20.000000" : 6688,
            "30.000000" : 9536,
            "40.000000" : 11968,
            "50.000000" : 14144,
            "60.000000" : 16512,
            "70.000000" : 18560,
            "80.000000" : 20352,
            "90.000000" : 22912,
            "95.000000" : 23424,
            "99.000000" : 27008,
            "99.500000" : 29312,
            "99.900000" : 45312,
            "99.950000" : 58624,
            "99.990000" : 70144
          }
        },
        "lat_ns" : {
          "min" : 6122,
          "max" : 147336,
          "mean" : 16755.017869,
          "stddev" : 6618.416459,
          "N" : 2131459
        },
        "bw_min" : 151543,
        "bw_max" : 152997,
        "bw_agg" : 100.000000,
        "bw_mean" : 152264.642857,
        "bw_dev" : 377.366300,
        "bw_samples" : 14,
        "iops_min" : 303086,
        "iops_max" : 305994,
        "iops_mean" : 304529.357143,
        "iops_stddev" : 754.613463,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 41.871429,
      "sys_cpu" : 58.100000,
      "ctx" : 26,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.501066,
        "2" : 26.540462,
        "4" : 58.078777,
        "8" : 14.879695,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 87.483879,
        "8" : 12.516121,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.044148,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 6.314982,
        "10" : 29.727393,
        "20" : 43.166475,
        "50" : 20.663397,
        "100" : 0.081963,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=144155: Wed Oct 26 04:52:10 2022
  read: IOPS=304k, BW=149MiB/s (156MB/s)(1041MiB/7001msec)
    slat (nsec): min=1572, max=80341, avg=2723.38, stdev=926.51
    clat (nsec): min=320, max=145703, avg=14031.64, stdev=6669.05
     lat (usec): min=6, max=147, avg=16.76, stdev= 6.62
    clat percentiles (nsec):
     |  1.00th=[ 3376],  5.00th=[ 3504], 10.00th=[ 4832], 20.00th=[ 6688],
     | 30.00th=[ 9536], 40.00th=[11968], 50.00th=[14144], 60.00th=[16512],
     | 70.00th=[18560], 80.00th=[20352], 90.00th=[22912], 95.00th=[23424],
     | 99.00th=[27008], 99.50th=[29312], 99.90th=[45312], 99.95th=[58624],
     | 99.99th=[70144]
   bw (  KiB/s): min=151543, max=152997, per=100.00%, avg=152264.64, stdev=377.37, samples=14
   iops        : min=303086, max=305994, avg=304529.36, stdev=754.61, samples=14
  lat (nsec)   : 500=0.04%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=6.31%, 10=29.73%, 20=43.17%, 50=20.66%
  lat (usec)   : 100=0.08%, 250=0.01%
  cpu          : usr=41.87%, sys=58.10%, ctx=26, majf=0, minf=0
  IO depths    : 1=0.5%, 2=26.5%, 4=58.1%, 8=14.9%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=87.5%, 8=12.5%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2131455,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=1041MiB (1091MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752745,
  "timestamp_ms" : 1666752745825,
  "time" : "Wed Oct 26 04:52:25 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1052019200,
        "io_kbytes" : 1027362,
        "bw_bytes" : 150266990,
        "bw" : 146745,
        "iops" : 293489.072990,
        "runtime" : 7001,
        "total_ios" : 2054717,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1623,
          "max" : 44804,
          "mean" : 2930.171240,
          "stddev" : 1093.681472,
          "N" : 2054718
        },
        "clat_ns" : {
          "min" : 751,
          "max" : 124714,
          "mean" : 14597.804246,
          "stddev" : 6757.196878,
          "N" : 2054717,
          "percentile" : {
            "1.000000" : 3920,
            "5.000000" : 4048,
            "10.000000" : 5280,
            "20.000000" : 8256,
            "30.000000" : 10176,
            "40.000000" : 12096,
            "50.000000" : 14656,
            "60.000000" : 16768,
            "70.000000" : 18560,
            "80.000000" : 20096,
            "90.000000" : 24192,
            "95.000000" : 24448,
            "99.000000" : 27264,
            "99.500000" : 30336,
            "99.900000" : 42752,
            "99.950000" : 56576,
            "99.990000" : 69120
          }
        },
        "lat_ns" : {
          "min" : 6472,
          "max" : 126397,
          "mean" : 17527.975420,
          "stddev" : 6799.693481,
          "N" : 2054717
        },
        "bw_min" : 146440,
        "bw_max" : 147365,
        "bw_agg" : 100.000000,
        "bw_mean" : 146789.214286,
        "bw_dev" : 284.407935,
        "bw_samples" : 14,
        "iops_min" : 292880,
        "iops_max" : 294730,
        "iops_mean" : 293578.500000,
        "iops_stddev" : 568.809925,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.014286,
      "sys_cpu" : 60.957143,
      "ctx" : 27,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.407453,
        "2" : 26.810310,
        "4" : 57.959417,
        "8" : 14.822820,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 87.512790,
        "8" : 12.487210,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.029931
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 3.778185,
        "10" : 23.005261,
        "20" : 50.434439,
        "50" : 22.678257,
        "100" : 0.072613,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=144204: Wed Oct 26 04:52:25 2022
  read: IOPS=293k, BW=143MiB/s (150MB/s)(1003MiB/7001msec)
    slat (nsec): min=1623, max=44804, avg=2930.17, stdev=1093.68
    clat (nsec): min=751, max=124714, avg=14597.80, stdev=6757.20
     lat (usec): min=6, max=126, avg=17.53, stdev= 6.80
    clat percentiles (nsec):
     |  1.00th=[ 3920],  5.00th=[ 4048], 10.00th=[ 5280], 20.00th=[ 8256],
     | 30.00th=[10176], 40.00th=[12096], 50.00th=[14656], 60.00th=[16768],
     | 70.00th=[18560], 80.00th=[20096], 90.00th=[24192], 95.00th=[24448],
     | 99.00th=[27264], 99.50th=[30336], 99.90th=[42752], 99.95th=[56576],
     | 99.99th=[69120]
   bw (  KiB/s): min=146440, max=147365, per=100.00%, avg=146789.21, stdev=284.41, samples=14
   iops        : min=292880, max=294730, avg=293578.50, stdev=568.81, samples=14
  lat (nsec)   : 1000=0.03%
  lat (usec)   : 2=0.01%, 4=3.78%, 10=23.01%, 20=50.43%, 50=22.68%
  lat (usec)   : 100=0.07%, 250=0.01%
  cpu          : usr=39.01%, sys=60.96%, ctx=27, majf=0, minf=0
  IO depths    : 1=0.4%, 2=26.8%, 4=58.0%, 8=14.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=87.5%, 8=12.5%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2054717,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=1003MiB (1052MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752761,
  "timestamp_ms" : 1666752761025,
  "time" : "Wed Oct 26 04:52:41 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1073317888,
        "io_kbytes" : 1048162,
        "bw_bytes" : 153309225,
        "bw" : 149716,
        "iops" : 299431.366948,
        "runtime" : 7001,
        "total_ios" : 2096319,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1593,
          "max" : 80861,
          "mean" : 2771.613320,
          "stddev" : 938.858044,
          "N" : 2096320
        },
        "clat_ns" : {
          "min" : 732,
          "max" : 127409,
          "mean" : 14376.397846,
          "stddev" : 6697.479279,
          "N" : 2096324,
          "percentile" : {
            "1.000000" : 3792,
            "5.000000" : 3888,
            "10.000000" : 5152,
            "20.000000" : 7200,
            "30.000000" : 10048,
            "40.000000" : 11840,
            "50.000000" : 14528,
            "60.000000" : 16768,
            "70.000000" : 18304,
            "80.000000" : 19840,
            "90.000000" : 23680,
            "95.000000" : 24192,
            "99.000000" : 27264,
            "99.500000" : 29824,
            "99.900000" : 41728,
            "99.950000" : 56576,
            "99.990000" : 70144
          }
        },
        "lat_ns" : {
          "min" : 6502,
          "max" : 129052,
          "mean" : 17148.011537,
          "stddev" : 6676.946354,
          "N" : 2096324
        },
        "bw_min" : 149307,
        "bw_max" : 150245,
        "bw_agg" : 100.000000,
        "bw_mean" : 149750.857143,
        "bw_dev" : 298.817891,
        "bw_samples" : 14,
        "iops_min" : 298614,
        "iops_max" : 300490,
        "iops_mean" : 299501.714286,
        "iops_stddev" : 597.243338,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.985714,
      "sys_cpu" : 59.985714,
      "ctx" : 25,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.448739,
        "2" : 26.840428,
        "4" : 57.885036,
        "8" : 14.825797,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 87.522870,
        "8" : 12.477130,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.039975
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 6.707996,
        "10" : 23.033327,
        "20" : 51.503469,
        "50" : 18.641581,
        "100" : 0.071936,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=144253: Wed Oct 26 04:52:41 2022
  read: IOPS=299k, BW=146MiB/s (153MB/s)(1024MiB/7001msec)
    slat (nsec): min=1593, max=80861, avg=2771.61, stdev=938.86
    clat (nsec): min=732, max=127409, avg=14376.40, stdev=6697.48
     lat (usec): min=6, max=129, avg=17.15, stdev= 6.68
    clat percentiles (nsec):
     |  1.00th=[ 3792],  5.00th=[ 3888], 10.00th=[ 5152], 20.00th=[ 7200],
     | 30.00th=[10048], 40.00th=[11840], 50.00th=[14528], 60.00th=[16768],
     | 70.00th=[18304], 80.00th=[19840], 90.00th=[23680], 95.00th=[24192],
     | 99.00th=[27264], 99.50th=[29824], 99.90th=[41728], 99.95th=[56576],
     | 99.99th=[70144]
   bw (  KiB/s): min=149307, max=150245, per=100.00%, avg=149750.86, stdev=298.82, samples=14
   iops        : min=298614, max=300490, avg=299501.71, stdev=597.24, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.04%
  lat (usec)   : 2=0.01%, 4=6.71%, 10=23.03%, 20=51.50%, 50=18.64%
  lat (usec)   : 100=0.07%, 250=0.01%
  cpu          : usr=39.99%, sys=59.99%, ctx=25, majf=0, minf=0
  IO depths    : 1=0.4%, 2=26.8%, 4=57.9%, 8=14.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=87.5%, 8=12.5%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2096319,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=1024MiB (1073MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_8_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752776,
  "timestamp_ms" : 1666752776202,
  "time" : "Wed Oct 26 04:52:56 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1060783104,
        "io_kbytes" : 1035921,
        "bw_bytes" : 151518797,
        "bw" : 147967,
        "iops" : 295934.152264,
        "runtime" : 7001,
        "total_ios" : 2071835,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1623,
          "max" : 49042,
          "mean" : 2860.583254,
          "stddev" : 1041.203387,
          "N" : 2071836
        },
        "clat_ns" : {
          "min" : 762,
          "max" : 127149,
          "mean" : 14498.881318,
          "stddev" : 6774.396456,
          "N" : 2071842,
          "percentile" : {
            "1.000000" : 3856,
            "5.000000" : 3984,
            "10.000000" : 5216,
            "20.000000" : 8256,
            "30.000000" : 10176,
            "40.000000" : 11968,
            "50.000000" : 14528,
            "60.000000" : 17280,
            "70.000000" : 18560,
            "80.000000" : 19840,
            "90.000000" : 23936,
            "95.000000" : 24448,
            "99.000000" : 27520,
            "99.500000" : 30592,
            "99.900000" : 46336,
            "99.950000" : 58624,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 6472,
          "max" : 128822,
          "mean" : 17359.464180,
          "stddev" : 6757.687788,
          "N" : 2071842
        },
        "bw_min" : 147584,
        "bw_max" : 148654,
        "bw_agg" : 100.000000,
        "bw_mean" : 148008.500000,
        "bw_dev" : 324.912710,
        "bw_samples" : 14,
        "iops_min" : 295168,
        "iops_max" : 297308,
        "iops_mean" : 296017.000000,
        "iops_stddev" : 650.166251,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.242857,
      "sys_cpu" : 60.728571,
      "ctx" : 24,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.416249,
        "2" : 26.862419,
        "4" : 57.902777,
        "8" : 14.818555,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 87.526588,
        "8" : 12.473412,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.035331
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 5.719471,
        "10" : 21.807528,
        "20" : 52.966718,
        "50" : 19.386679,
        "100" : 0.083260,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=144302: Wed Oct 26 04:52:56 2022
  read: IOPS=296k, BW=144MiB/s (152MB/s)(1012MiB/7001msec)
    slat (nsec): min=1623, max=49042, avg=2860.58, stdev=1041.20
    clat (nsec): min=762, max=127149, avg=14498.88, stdev=6774.40
     lat (usec): min=6, max=128, avg=17.36, stdev= 6.76
    clat percentiles (nsec):
     |  1.00th=[ 3856],  5.00th=[ 3984], 10.00th=[ 5216], 20.00th=[ 8256],
     | 30.00th=[10176], 40.00th=[11968], 50.00th=[14528], 60.00th=[17280],
     | 70.00th=[18560], 80.00th=[19840], 90.00th=[23936], 95.00th=[24448],
     | 99.00th=[27520], 99.50th=[30592], 99.90th=[46336], 99.95th=[58624],
     | 99.99th=[72192]
   bw (  KiB/s): min=147584, max=148654, per=100.00%, avg=148008.50, stdev=324.91, samples=14
   iops        : min=295168, max=297308, avg=296017.00, stdev=650.17, samples=14
  lat (nsec)   : 1000=0.04%
  lat (usec)   : 2=0.01%, 4=5.72%, 10=21.81%, 20=52.97%, 50=19.39%
  lat (usec)   : 100=0.08%, 250=0.01%
  cpu          : usr=39.24%, sys=60.73%, ctx=24, majf=0, minf=0
  IO depths    : 1=0.4%, 2=26.9%, 4=57.9%, 8=14.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=87.5%, 8=12.5%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2071835,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=144MiB/s (152MB/s), 144MiB/s-144MiB/s (152MB/s-152MB/s), io=1012MiB (1061MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751109,
  "timestamp_ms" : 1666751109430,
  "time" : "Wed Oct 26 04:25:09 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 554295808,
        "io_kbytes" : 541304,
        "bw_bytes" : 79173804,
        "bw" : 77318,
        "iops" : 154636.337666,
        "runtime" : 7001,
        "total_ios" : 1082609,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 160,
          "max" : 11602,
          "mean" : 177.716369,
          "stddev" : 44.536553,
          "N" : 1082609
        },
        "clat_ns" : {
          "min" : 5681,
          "max" : 127730,
          "mean" : 6024.580633,
          "stddev" : 1112.814196,
          "N" : 1082609,
          "percentile" : {
            "1.000000" : 5728,
            "5.000000" : 5792,
            "10.000000" : 5792,
            "20.000000" : 5792,
            "30.000000" : 5792,
            "40.000000" : 5856,
            "50.000000" : 5856,
            "60.000000" : 5920,
            "70.000000" : 5920,
            "80.000000" : 5920,
            "90.000000" : 6048,
            "95.000000" : 6112,
            "99.000000" : 11328,
            "99.500000" : 12608,
            "99.900000" : 17024,
            "99.950000" : 17024,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 5851,
          "max" : 127900,
          "mean" : 6202.297002,
          "stddev" : 1114.031677,
          "N" : 1082609
        },
        "bw_min" : 77106,
        "bw_max" : 77614,
        "bw_agg" : 100.000000,
        "bw_mean" : 77336.642857,
        "bw_dev" : 146.503141,
        "bw_samples" : 14,
        "iops_min" : 154212,
        "iops_max" : 155228,
        "iops_mean" : 154673.285714,
        "iops_stddev" : 293.006282,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 51.814286,
      "sys_cpu" : 48.157143,
      "ctx" : 21,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.603836,
        "20" : 1.381108,
        "50" : 0.010000,
        "100" : 0.012378,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=139184: Wed Oct 26 04:25:09 2022
  read: IOPS=155k, BW=75.5MiB/s (79.2MB/s)(529MiB/7001msec)
    slat (nsec): min=160, max=11602, avg=177.72, stdev=44.54
    clat (usec): min=5, max=127, avg= 6.02, stdev= 1.11
     lat (usec): min=5, max=127, avg= 6.20, stdev= 1.11
    clat percentiles (nsec):
     |  1.00th=[ 5728],  5.00th=[ 5792], 10.00th=[ 5792], 20.00th=[ 5792],
     | 30.00th=[ 5792], 40.00th=[ 5856], 50.00th=[ 5856], 60.00th=[ 5920],
     | 70.00th=[ 5920], 80.00th=[ 5920], 90.00th=[ 6048], 95.00th=[ 6112],
     | 99.00th=[11328], 99.50th=[12608], 99.90th=[17024], 99.95th=[17024],
     | 99.99th=[58112]
   bw (  KiB/s): min=77106, max=77614, per=100.00%, avg=77336.64, stdev=146.50, samples=14
   iops        : min=154212, max=155228, avg=154673.29, stdev=293.01, samples=14
  lat (usec)   : 10=98.60%, 20=1.38%, 50=0.01%, 100=0.01%, 250=0.01%
  cpu          : usr=51.81%, sys=48.16%, ctx=21, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1082609,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=75.5MiB/s (79.2MB/s), 75.5MiB/s-75.5MiB/s (79.2MB/s-79.2MB/s), io=529MiB (554MB), run=7001-7001msec

$ cat /tmp/fio-output_512_8_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752789,
  "timestamp_ms" : 1666752789979,
  "time" : "Wed Oct 26 04:53:09 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1014642688,
        "io_kbytes" : 990862,
        "bw_bytes" : 144928251,
        "bw" : 141531,
        "iops" : 283061.848307,
        "runtime" : 7001,
        "total_ios" : 1981716,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1472,
          "max" : 32762,
          "mean" : 2358.838814,
          "stddev" : 1033.707278,
          "N" : 1981717
        },
        "clat_ns" : {
          "min" : 6823,
          "max" : 153127,
          "mean" : 25221.540123,
          "stddev" : 3166.343719,
          "N" : 1981723,
          "percentile" : {
            "1.000000" : 18304,
            "5.000000" : 23168,
            "10.000000" : 23424,
            "20.000000" : 23424,
            "30.000000" : 23680,
            "40.000000" : 23680,
            "50.000000" : 24960,
            "60.000000" : 25984,
            "70.000000" : 26240,
            "80.000000" : 26240,
            "90.000000" : 26496,
            "95.000000" : 29568,
            "99.000000" : 38144,
            "99.500000" : 42752,
            "99.900000" : 51968,
            "99.950000" : 64768,
            "99.990000" : 78336
          }
        },
        "lat_ns" : {
          "min" : 10910,
          "max" : 154660,
          "mean" : 27580.380426,
          "stddev" : 2971.138538,
          "N" : 1981723
        },
        "bw_min" : 141148,
        "bw_max" : 141868,
        "bw_agg" : 100.000000,
        "bw_mean" : 141569.500000,
        "bw_dev" : 202.955755,
        "bw_samples" : 14,
        "iops_min" : 282296,
        "iops_max" : 283736,
        "iops_mean" : 283139.142857,
        "iops_stddev" : 405.859175,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 36.985714,
      "sys_cpu" : 62.000000,
      "ctx" : 9204,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999975,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 1.889827,
        "50" : 97.980891,
        "100" : 0.126507,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=144341: Wed Oct 26 04:53:09 2022
  read: IOPS=283k, BW=138MiB/s (145MB/s)(968MiB/7001msec)
    slat (nsec): min=1472, max=32762, avg=2358.84, stdev=1033.71
    clat (usec): min=6, max=153, avg=25.22, stdev= 3.17
     lat (usec): min=10, max=154, avg=27.58, stdev= 2.97
    clat percentiles (nsec):
     |  1.00th=[18304],  5.00th=[23168], 10.00th=[23424], 20.00th=[23424],
     | 30.00th=[23680], 40.00th=[23680], 50.00th=[24960], 60.00th=[25984],
     | 70.00th=[26240], 80.00th=[26240], 90.00th=[26496], 95.00th=[29568],
     | 99.00th=[38144], 99.50th=[42752], 99.90th=[51968], 99.95th=[64768],
     | 99.99th=[78336]
   bw (  KiB/s): min=141148, max=141868, per=100.00%, avg=141569.50, stdev=202.96, samples=14
   iops        : min=282296, max=283736, avg=283139.14, stdev=405.86, samples=14
  lat (usec)   : 10=0.01%, 20=1.89%, 50=97.98%, 100=0.13%, 250=0.01%
  cpu          : usr=36.99%, sys=62.00%, ctx=9204, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1981716,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=968MiB (1015MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752803,
  "timestamp_ms" : 1666752803651,
  "time" : "Wed Oct 26 04:53:23 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1012346368,
        "io_kbytes" : 988619,
        "bw_bytes" : 144600252,
        "bw" : 141211,
        "iops" : 282421.368376,
        "runtime" : 7001,
        "total_ios" : 1977232,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1482,
          "max" : 54362,
          "mean" : 2372.692545,
          "stddev" : 1039.385184,
          "N" : 1977232
        },
        "clat_ns" : {
          "min" : 4679,
          "max" : 146735,
          "mean" : 25274.840964,
          "stddev" : 3228.285563,
          "N" : 1977239,
          "percentile" : {
            "1.000000" : 18304,
            "5.000000" : 23168,
            "10.000000" : 23424,
            "20.000000" : 23424,
            "30.000000" : 23680,
            "40.000000" : 23936,
            "50.000000" : 24960,
            "60.000000" : 25984,
            "70.000000" : 26240,
            "80.000000" : 26240,
            "90.000000" : 26496,
            "95.000000" : 30080,
            "99.000000" : 38656,
            "99.500000" : 42752,
            "99.900000" : 53504,
            "99.950000" : 66048,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 6211,
          "max" : 151143,
          "mean" : 27647.531768,
          "stddev" : 3048.297465,
          "N" : 1977239
        },
        "bw_min" : 140956,
        "bw_max" : 141621,
        "bw_agg" : 100.000000,
        "bw_mean" : 141248.214286,
        "bw_dev" : 159.472293,
        "bw_samples" : 14,
        "iops_min" : 281912,
        "iops_max" : 283242,
        "iops_mean" : 282496.500000,
        "iops_stddev" : 318.744351,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 38.000000,
      "sys_cpu" : 60.957143,
      "ctx" : 9034,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999975,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 1.892090,
        "50" : 97.959976,
        "100" : 0.144242,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=144380: Wed Oct 26 04:53:23 2022
  read: IOPS=282k, BW=138MiB/s (145MB/s)(965MiB/7001msec)
    slat (nsec): min=1482, max=54362, avg=2372.69, stdev=1039.39
    clat (usec): min=4, max=146, avg=25.27, stdev= 3.23
     lat (usec): min=6, max=151, avg=27.65, stdev= 3.05
    clat percentiles (nsec):
     |  1.00th=[18304],  5.00th=[23168], 10.00th=[23424], 20.00th=[23424],
     | 30.00th=[23680], 40.00th=[23936], 50.00th=[24960], 60.00th=[25984],
     | 70.00th=[26240], 80.00th=[26240], 90.00th=[26496], 95.00th=[30080],
     | 99.00th=[38656], 99.50th=[42752], 99.90th=[53504], 99.95th=[66048],
     | 99.99th=[79360]
   bw (  KiB/s): min=140956, max=141621, per=100.00%, avg=141248.21, stdev=159.47, samples=14
   iops        : min=281912, max=283242, avg=282496.50, stdev=318.74, samples=14
  lat (usec)   : 10=0.01%, 20=1.89%, 50=97.96%, 100=0.14%, 250=0.01%
  cpu          : usr=38.00%, sys=60.96%, ctx=9034, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1977232,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=965MiB (1012MB), run=7001-7001msec

$ rm /tmp/fio-output_512_8_libaio-xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_8_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_8_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666752817,
  "timestamp_ms" : 1666752817296,
  "time" : "Wed Oct 26 04:53:37 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 1011662336,
        "io_kbytes" : 987951,
        "bw_bytes" : 144502547,
        "bw" : 141115,
        "iops" : 282230.395658,
        "runtime" : 7001,
        "total_ios" : 1975895,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1482,
          "max" : 25537,
          "mean" : 2298.559704,
          "stddev" : 931.379939,
          "N" : 1975896
        },
        "clat_ns" : {
          "min" : 4278,
          "max" : 139502,
          "mean" : 25292.891398,
          "stddev" : 3159.053006,
          "N" : 1975902,
          "percentile" : {
            "1.000000" : 18304,
            "5.000000" : 23168,
            "10.000000" : 23424,
            "20.000000" : 23680,
            "30.000000" : 23680,
            "40.000000" : 23936,
            "50.000000" : 24960,
            "60.000000" : 25984,
            "70.000000" : 26240,
            "80.000000" : 26496,
            "90.000000" : 26496,
            "95.000000" : 29824,
            "99.000000" : 38144,
            "99.500000" : 42752,
            "99.900000" : 52480,
            "99.950000" : 66048,
            "99.990000" : 78336
          }
        },
        "lat_ns" : {
          "min" : 5811,
          "max" : 141034,
          "mean" : 27591.452594,
          "stddev" : 2976.337309,
          "N" : 1975902
        },
        "bw_min" : 140763,
        "bw_max" : 141655,
        "bw_agg" : 100.000000,
        "bw_mean" : 141148.214286,
        "bw_dev" : 223.325281,
        "bw_samples" : 14,
        "iops_min" : 281526,
        "iops_max" : 283310,
        "iops_mean" : 282296.428571,
        "iops_stddev" : 446.650563,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.814286,
      "sys_cpu" : 59.185714,
      "ctx" : 8781,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999975,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 1.898886,
        "50" : 97.964619,
        "100" : 0.134015,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=144420: Wed Oct 26 04:53:37 2022
  read: IOPS=282k, BW=138MiB/s (145MB/s)(965MiB/7001msec)
    slat (nsec): min=1482, max=25537, avg=2298.56, stdev=931.38
    clat (usec): min=4, max=139, avg=25.29, stdev= 3.16
     lat (usec): min=5, max=141, avg=27.59, stdev= 2.98
    clat percentiles (nsec):
     |  1.00th=[18304],  5.00th=[23168], 10.00th=[23424], 20.00th=[23680],
     | 30.00th=[23680], 40.00th=[23936], 50.00th=[24960], 60.00th=[25984],
     | 70.00th=[26240], 80.00th=[26496], 90.00th=[26496], 95.00th=[29824],
     | 99.00th=[38144], 99.50th=[42752], 99.90th=[52480], 99.95th=[66048],
     | 99.99th=[78336]
   bw (  KiB/s): min=140763, max=141655, per=100.00%, avg=141148.21, stdev=223.33, samples=14
   iops        : min=281526, max=283310, avg=282296.43, stdev=446.65, samples=14
  lat (usec)   : 10=0.01%, 20=1.90%, 50=97.96%, 100=0.13%, 250=0.01%
  cpu          : usr=39.81%, sys=59.19%, ctx=8781, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1975895,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=965MiB (1012MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751124,
  "timestamp_ms" : 1666751124416,
  "time" : "Wed Oct 26 04:25:24 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 554997760,
        "io_kbytes" : 541990,
        "bw_bytes" : 79285394,
        "bw" : 77427,
        "iops" : 154854.285714,
        "runtime" : 7000,
        "total_ios" : 1083980,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 160,
          "max" : 12163,
          "mean" : 182.966398,
          "stddev" : 38.361146,
          "N" : 1083980
        },
        "clat_ns" : {
          "min" : 5690,
          "max" : 79259,
          "mean" : 6045.423559,
          "stddev" : 1095.010971,
          "N" : 1083980,
          "percentile" : {
            "1.000000" : 5792,
            "5.000000" : 5792,
            "10.000000" : 5792,
            "20.000000" : 5856,
            "30.000000" : 5856,
            "40.000000" : 5856,
            "50.000000" : 5920,
            "60.000000" : 5920,
            "70.000000" : 5920,
            "80.000000" : 5984,
            "90.000000" : 6048,
            "95.000000" : 6176,
            "99.000000" : 11456,
            "99.500000" : 12096,
            "99.900000" : 17024,
            "99.950000" : 17024,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 5871,
          "max" : 79429,
          "mean" : 6228.389957,
          "stddev" : 1095.955735,
          "N" : 1083980
        },
        "bw_min" : 77263,
        "bw_max" : 77571,
        "bw_agg" : 100.000000,
        "bw_mean" : 77436.857143,
        "bw_dev" : 79.987362,
        "bw_samples" : 14,
        "iops_min" : 154526,
        "iops_max" : 155142,
        "iops_mean" : 154873.785714,
        "iops_stddev" : 160.033736,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 48.900000,
      "sys_cpu" : 51.071429,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.581247,
        "20" : 1.403808,
        "50" : 0.010000,
        "100" : 0.013100,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=139233: Wed Oct 26 04:25:24 2022
  read: IOPS=155k, BW=75.6MiB/s (79.3MB/s)(529MiB/7000msec)
    slat (nsec): min=160, max=12163, avg=182.97, stdev=38.36
    clat (nsec): min=5690, max=79259, avg=6045.42, stdev=1095.01
     lat (nsec): min=5871, max=79429, avg=6228.39, stdev=1095.96
    clat percentiles (nsec):
     |  1.00th=[ 5792],  5.00th=[ 5792], 10.00th=[ 5792], 20.00th=[ 5856],
     | 30.00th=[ 5856], 40.00th=[ 5856], 50.00th=[ 5920], 60.00th=[ 5920],
     | 70.00th=[ 5920], 80.00th=[ 5984], 90.00th=[ 6048], 95.00th=[ 6176],
     | 99.00th=[11456], 99.50th=[12096], 99.90th=[17024], 99.95th=[17024],
     | 99.99th=[58112]
   bw (  KiB/s): min=77263, max=77571, per=100.00%, avg=77436.86, stdev=79.99, samples=14
   iops        : min=154526, max=155142, avg=154873.79, stdev=160.03, samples=14
  lat (usec)   : 10=98.58%, 20=1.40%, 50=0.01%, 100=0.01%
  cpu          : usr=48.90%, sys=51.07%, ctx=23, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1083980,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=75.6MiB/s (79.3MB/s), 75.6MiB/s-75.6MiB/s (79.3MB/s-79.3MB/s), io=529MiB (555MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_io_uring-bdev_uring_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_uring_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751139,
  "timestamp_ms" : 1666751139437,
  "time" : "Wed Oct 26 04:25:39 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 556163072,
        "io_kbytes" : 543128,
        "bw_bytes" : 79440518,
        "bw" : 77578,
        "iops" : 155157.263248,
        "runtime" : 7001,
        "total_ios" : 1086256,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 160,
          "max" : 11702,
          "mean" : 182.538771,
          "stddev" : 44.114911,
          "N" : 1086256
        },
        "clat_ns" : {
          "min" : 5681,
          "max" : 114124,
          "mean" : 6027.226244,
          "stddev" : 1126.443798,
          "N" : 1086256,
          "percentile" : {
            "1.000000" : 5728,
            "5.000000" : 5792,
            "10.000000" : 5792,
            "20.000000" : 5792,
            "30.000000" : 5792,
            "40.000000" : 5856,
            "50.000000" : 5856,
            "60.000000" : 5920,
            "70.000000" : 5920,
            "80.000000" : 5920,
            "90.000000" : 6048,
            "95.000000" : 6112,
            "99.000000" : 11456,
            "99.500000" : 12736,
            "99.900000" : 17024,
            "99.950000" : 17024,
            "99.990000" : 58624
          }
        },
        "lat_ns" : {
          "min" : 5861,
          "max" : 114305,
          "mean" : 6209.765015,
          "stddev" : 1127.684770,
          "N" : 1086256
        },
        "bw_min" : 77442,
        "bw_max" : 77945,
        "bw_agg" : 100.000000,
        "bw_mean" : 77599.142857,
        "bw_dev" : 130.531374,
        "bw_samples" : 14,
        "iops_min" : 154884,
        "iops_max" : 155890,
        "iops_mean" : 155198.357143,
        "iops_stddev" : 261.172388,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 47.071429,
      "sys_cpu" : 52.900000,
      "ctx" : 18,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 98.564611,
        "20" : 1.419371,
        "50" : 0.010000,
        "100" : 0.013809,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=139283: Wed Oct 26 04:25:39 2022
  read: IOPS=155k, BW=75.8MiB/s (79.4MB/s)(530MiB/7001msec)
    slat (nsec): min=160, max=11702, avg=182.54, stdev=44.11
    clat (usec): min=5, max=114, avg= 6.03, stdev= 1.13
     lat (usec): min=5, max=114, avg= 6.21, stdev= 1.13
    clat percentiles (nsec):
     |  1.00th=[ 5728],  5.00th=[ 5792], 10.00th=[ 5792], 20.00th=[ 5792],
     | 30.00th=[ 5792], 40.00th=[ 5856], 50.00th=[ 5856], 60.00th=[ 5920],
     | 70.00th=[ 5920], 80.00th=[ 5920], 90.00th=[ 6048], 95.00th=[ 6112],
     | 99.00th=[11456], 99.50th=[12736], 99.90th=[17024], 99.95th=[17024],
     | 99.99th=[58624]
   bw (  KiB/s): min=77442, max=77945, per=100.00%, avg=77599.14, stdev=130.53, samples=14
   iops        : min=154884, max=155890, avg=155198.36, stdev=261.17, samples=14
  lat (usec)   : 10=98.56%, 20=1.42%, 50=0.01%, 100=0.01%, 250=0.01%
  cpu          : usr=47.07%, sys=52.90%, ctx=18, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1086256,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=75.8MiB/s (79.4MB/s), 75.8MiB/s-75.8MiB/s (79.4MB/s-79.4MB/s), io=530MiB (556MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751154,
  "timestamp_ms" : 1666751154404,
  "time" : "Wed Oct 26 04:25:54 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 579297280,
        "io_kbytes" : 565720,
        "bw_bytes" : 82744933,
        "bw" : 80805,
        "iops" : 161611.055563,
        "runtime" : 7001,
        "total_ios" : 1131439,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1573,
          "max" : 22613,
          "mean" : 1629.937999,
          "stddev" : 110.064556,
          "N" : 1131440
        },
        "clat_ns" : {
          "min" : 231,
          "max" : 100949,
          "mean" : 4328.043267,
          "stddev" : 1152.033943,
          "N" : 1131439,
          "percentile" : {
            "1.000000" : 4048,
            "5.000000" : 4080,
            "10.000000" : 4080,
            "20.000000" : 4128,
            "30.000000" : 4128,
            "40.000000" : 4128,
            "50.000000" : 4192,
            "60.000000" : 4192,
            "70.000000" : 4256,
            "80.000000" : 4256,
            "90.000000" : 4320,
            "95.000000" : 4448,
            "99.000000" : 10048,
            "99.500000" : 10688,
            "99.900000" : 15424,
            "99.950000" : 15552,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5620,
          "max" : 102592,
          "mean" : 5957.981281,
          "stddev" : 1158.044008,
          "N" : 1131439
        },
        "bw_min" : 80689,
        "bw_max" : 80965,
        "bw_agg" : 100.000000,
        "bw_mean" : 80824.285714,
        "bw_dev" : 84.986360,
        "bw_samples" : 14,
        "iops_min" : 161378,
        "iops_max" : 161930,
        "iops_mean" : 161648.642857,
        "iops_stddev" : 169.934651,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.557143,
      "sys_cpu" : 89.414286,
      "ctx" : 24,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 98.957080,
        "20" : 1.019763,
        "50" : 0.010000,
        "100" : 0.014495,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139333: Wed Oct 26 04:25:54 2022
  read: IOPS=162k, BW=78.9MiB/s (82.7MB/s)(552MiB/7001msec)
    slat (nsec): min=1573, max=22613, avg=1629.94, stdev=110.06
    clat (nsec): min=231, max=100949, avg=4328.04, stdev=1152.03
     lat (usec): min=5, max=102, avg= 5.96, stdev= 1.16
    clat percentiles (nsec):
     |  1.00th=[ 4048],  5.00th=[ 4080], 10.00th=[ 4080], 20.00th=[ 4128],
     | 30.00th=[ 4128], 40.00th=[ 4128], 50.00th=[ 4192], 60.00th=[ 4192],
     | 70.00th=[ 4256], 80.00th=[ 4256], 90.00th=[ 4320], 95.00th=[ 4448],
     | 99.00th=[10048], 99.50th=[10688], 99.90th=[15424], 99.95th=[15552],
     | 99.99th=[57088]
   bw (  KiB/s): min=80689, max=80965, per=100.00%, avg=80824.29, stdev=84.99, samples=14
   iops        : min=161378, max=161930, avg=161648.64, stdev=169.93, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=98.96%, 20=1.02%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=10.56%, sys=89.41%, ctx=24, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1131439,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=78.9MiB/s (82.7MB/s), 78.9MiB/s-78.9MiB/s (82.7MB/s-82.7MB/s), io=552MiB (579MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751169,
  "timestamp_ms" : 1666751169479,
  "time" : "Wed Oct 26 04:26:09 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 578703872,
        "io_kbytes" : 565140,
        "bw_bytes" : 82671981,
        "bw" : 80734,
        "iops" : 161468.714286,
        "runtime" : 7000,
        "total_ios" : 1130281,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1563,
          "max" : 15979,
          "mean" : 1627.234484,
          "stddev" : 111.403997,
          "N" : 1130281
        },
        "clat_ns" : {
          "min" : 230,
          "max" : 75993,
          "mean" : 4333.185862,
          "stddev" : 1142.444879,
          "N" : 1130281,
          "percentile" : {
            "1.000000" : 4080,
            "5.000000" : 4080,
            "10.000000" : 4128,
            "20.000000" : 4128,
            "30.000000" : 4128,
            "40.000000" : 4192,
            "50.000000" : 4192,
            "60.000000" : 4192,
            "70.000000" : 4256,
            "80.000000" : 4256,
            "90.000000" : 4320,
            "95.000000" : 4448,
            "99.000000" : 10048,
            "99.500000" : 11200,
            "99.900000" : 15424,
            "99.950000" : 15552,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5630,
          "max" : 77626,
          "mean" : 5960.420346,
          "stddev" : 1148.508228,
          "N" : 1130281
        },
        "bw_min" : 80670,
        "bw_max" : 80870,
        "bw_agg" : 100.000000,
        "bw_mean" : 80741.785714,
        "bw_dev" : 54.574125,
        "bw_samples" : 14,
        "iops_min" : 161340,
        "iops_max" : 161740,
        "iops_mean" : 161483.571429,
        "iops_stddev" : 109.148251,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.014286,
      "sys_cpu" : 88.957143,
      "ctx" : 20,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 98.965390,
        "20" : 1.013199,
        "50" : 0.010000,
        "100" : 0.012740,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139382: Wed Oct 26 04:26:09 2022
  read: IOPS=161k, BW=78.8MiB/s (82.7MB/s)(552MiB/7000msec)
    slat (nsec): min=1563, max=15979, avg=1627.23, stdev=111.40
    clat (nsec): min=230, max=75993, avg=4333.19, stdev=1142.44
     lat (nsec): min=5630, max=77626, avg=5960.42, stdev=1148.51
    clat percentiles (nsec):
     |  1.00th=[ 4080],  5.00th=[ 4080], 10.00th=[ 4128], 20.00th=[ 4128],
     | 30.00th=[ 4128], 40.00th=[ 4192], 50.00th=[ 4192], 60.00th=[ 4192],
     | 70.00th=[ 4256], 80.00th=[ 4256], 90.00th=[ 4320], 95.00th=[ 4448],
     | 99.00th=[10048], 99.50th=[11200], 99.90th=[15424], 99.95th=[15552],
     | 99.99th=[57088]
   bw (  KiB/s): min=80670, max=80870, per=100.00%, avg=80741.79, stdev=54.57, samples=14
   iops        : min=161340, max=161740, avg=161483.57, stdev=109.15, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=98.97%, 20=1.01%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=11.01%, sys=88.96%, ctx=20, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1130281,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=78.8MiB/s (82.7MB/s), 78.8MiB/s-78.8MiB/s (82.7MB/s-82.7MB/s), io=552MiB (579MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751184,
  "timestamp_ms" : 1666751184500,
  "time" : "Wed Oct 26 04:26:24 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 576747520,
        "io_kbytes" : 563230,
        "bw_bytes" : 82380734,
        "bw" : 80449,
        "iops" : 160899.871447,
        "runtime" : 7001,
        "total_ios" : 1126460,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1563,
          "max" : 15619,
          "mean" : 1623.509082,
          "stddev" : 104.723698,
          "N" : 1126460
        },
        "clat_ns" : {
          "min" : 241,
          "max" : 77606,
          "mean" : 4359.351258,
          "stddev" : 1122.867630,
          "N" : 1126460,
          "percentile" : {
            "1.000000" : 4128,
            "5.000000" : 4128,
            "10.000000" : 4128,
            "20.000000" : 4128,
            "30.000000" : 4128,
            "40.000000" : 4192,
            "50.000000" : 4192,
            "60.000000" : 4256,
            "70.000000" : 4256,
            "80.000000" : 4320,
            "90.000000" : 4384,
            "95.000000" : 4448,
            "99.000000" : 10048,
            "99.500000" : 10688,
            "99.900000" : 15424,
            "99.950000" : 15552,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5650,
          "max" : 79209,
          "mean" : 5982.860339,
          "stddev" : 1128.826196,
          "N" : 1126460
        },
        "bw_min" : 80316,
        "bw_max" : 80630,
        "bw_agg" : 100.000000,
        "bw_mean" : 80486.615385,
        "bw_dev" : 102.524419,
        "bw_samples" : 13,
        "iops_min" : 160632,
        "iops_max" : 161260,
        "iops_mean" : 160973.230769,
        "iops_stddev" : 205.048837,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.328571,
      "sys_cpu" : 89.642857,
      "ctx" : 21,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 98.971468,
        "20" : 1.008291,
        "50" : 0.010000,
        "100" : 0.013050,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139431: Wed Oct 26 04:26:24 2022
  read: IOPS=161k, BW=78.6MiB/s (82.4MB/s)(550MiB/7001msec)
    slat (nsec): min=1563, max=15619, avg=1623.51, stdev=104.72
    clat (nsec): min=241, max=77606, avg=4359.35, stdev=1122.87
     lat (nsec): min=5650, max=79209, avg=5982.86, stdev=1128.83
    clat percentiles (nsec):
     |  1.00th=[ 4128],  5.00th=[ 4128], 10.00th=[ 4128], 20.00th=[ 4128],
     | 30.00th=[ 4128], 40.00th=[ 4192], 50.00th=[ 4192], 60.00th=[ 4256],
     | 70.00th=[ 4256], 80.00th=[ 4320], 90.00th=[ 4384], 95.00th=[ 4448],
     | 99.00th=[10048], 99.50th=[10688], 99.90th=[15424], 99.95th=[15552],
     | 99.99th=[57088]
   bw (  KiB/s): min=80316, max=80630, per=100.00%, avg=80486.62, stdev=102.52, samples=13
   iops        : min=160632, max=161260, avg=160973.23, stdev=205.05, samples=13
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=98.97%, 20=1.01%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=10.33%, sys=89.64%, ctx=21, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1126460,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=78.6MiB/s (82.4MB/s), 78.6MiB/s-78.6MiB/s (82.4MB/s-82.4MB/s), io=550MiB (577MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-xnvme_0.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751198,
  "timestamp_ms" : 1666751198179,
  "time" : "Wed Oct 26 04:26:38 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 278675456,
        "io_kbytes" : 272144,
        "bw_bytes" : 39805092,
        "bw" : 38872,
        "iops" : 77744.179403,
        "runtime" : 7001,
        "total_ios" : 544287,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2585,
          "max" : 41207,
          "mean" : 2875.072484,
          "stddev" : 249.042554,
          "N" : 544288
        },
        "clat_ns" : {
          "min" : 301,
          "max" : 155822,
          "mean" : 9423.841889,
          "stddev" : 1236.552305,
          "N" : 544287,
          "percentile" : {
            "1.000000" : 9024,
            "5.000000" : 9024,
            "10.000000" : 9152,
            "20.000000" : 9152,
            "30.000000" : 9280,
            "40.000000" : 9280,
            "50.000000" : 9280,
            "60.000000" : 9280,
            "70.000000" : 9280,
            "80.000000" : 9408,
            "90.000000" : 9408,
            "95.000000" : 9664,
            "99.000000" : 15168,
            "99.500000" : 15808,
            "99.900000" : 19840,
            "99.950000" : 26752,
            "99.990000" : 55552
          }
        },
        "lat_ns" : {
          "min" : 9348,
          "max" : 158678,
          "mean" : 12298.914446,
          "stddev" : 1258.957845,
          "N" : 544287
        },
        "bw_min" : 38691,
        "bw_max" : 39063,
        "bw_agg" : 100.000000,
        "bw_mean" : 38882.357143,
        "bw_dev" : 110.504860,
        "bw_samples" : 14,
        "iops_min" : 77382,
        "iops_max" : 78126,
        "iops_mean" : 77764.714286,
        "iops_stddev" : 221.009721,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.557143,
      "sys_cpu" : 47.928571,
      "ctx" : 544144,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.011942,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.766412,
        "20" : 3.138785,
        "50" : 0.065958,
        "100" : 0.016535,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=139470: Wed Oct 26 04:26:38 2022
  read: IOPS=77.7k, BW=38.0MiB/s (39.8MB/s)(266MiB/7001msec)
    slat (nsec): min=2585, max=41207, avg=2875.07, stdev=249.04
    clat (nsec): min=301, max=155822, avg=9423.84, stdev=1236.55
     lat (usec): min=9, max=158, avg=12.30, stdev= 1.26
    clat percentiles (nsec):
     |  1.00th=[ 9024],  5.00th=[ 9024], 10.00th=[ 9152], 20.00th=[ 9152],
     | 30.00th=[ 9280], 40.00th=[ 9280], 50.00th=[ 9280], 60.00th=[ 9280],
     | 70.00th=[ 9280], 80.00th=[ 9408], 90.00th=[ 9408], 95.00th=[ 9664],
     | 99.00th=[15168], 99.50th=[15808], 99.90th=[19840], 99.95th=[26752],
     | 99.99th=[55552]
   bw (  KiB/s): min=38691, max=39063, per=100.00%, avg=38882.36, stdev=110.50, samples=14
   iops        : min=77382, max=78126, avg=77764.71, stdev=221.01, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 10=96.77%, 20=3.14%, 50=0.07%, 100=0.02%, 250=0.01%
  cpu          : usr=13.56%, sys=47.93%, ctx=544144, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=544287,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=38.0MiB/s (39.8MB/s), 38.0MiB/s-38.0MiB/s (39.8MB/s-39.8MB/s), io=266MiB (279MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-xnvme_1.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751211,
  "timestamp_ms" : 1666751211808,
  "time" : "Wed Oct 26 04:26:51 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 277933056,
        "io_kbytes" : 271419,
        "bw_bytes" : 39699050,
        "bw" : 38768,
        "iops" : 77537.208970,
        "runtime" : 7001,
        "total_ios" : 542838,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2625,
          "max" : 47760,
          "mean" : 2836.992208,
          "stddev" : 284.296938,
          "N" : 542838
        },
        "clat_ns" : {
          "min" : 311,
          "max" : 110006,
          "mean" : 9445.447319,
          "stddev" : 1220.518444,
          "N" : 542838,
          "percentile" : {
            "1.000000" : 9024,
            "5.000000" : 9152,
            "10.000000" : 9152,
            "20.000000" : 9152,
            "30.000000" : 9280,
            "40.000000" : 9280,
            "50.000000" : 9280,
            "60.000000" : 9280,
            "70.000000" : 9408,
            "80.000000" : 9408,
            "90.000000" : 9408,
            "95.000000" : 9664,
            "99.000000" : 15168,
            "99.500000" : 15808,
            "99.900000" : 20096,
            "99.950000" : 27264,
            "99.990000" : 56064
          }
        },
        "lat_ns" : {
          "min" : 9437,
          "max" : 112812,
          "mean" : 12282.439527,
          "stddev" : 1251.244279,
          "N" : 542838
        },
        "bw_min" : 38539,
        "bw_max" : 38959,
        "bw_agg" : 100.000000,
        "bw_mean" : 38780.285714,
        "bw_dev" : 105.847807,
        "bw_samples" : 14,
        "iops_min" : 77078,
        "iops_max" : 77918,
        "iops_mean" : 77560.642857,
        "iops_stddev" : 211.751526,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.314286,
      "sys_cpu" : 48.014286,
      "ctx" : 542695,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010685,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.858547,
        "20" : 3.030923,
        "50" : 0.083819,
        "100" : 0.015658,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=139510: Wed Oct 26 04:26:51 2022
  read: IOPS=77.5k, BW=37.9MiB/s (39.7MB/s)(265MiB/7001msec)
    slat (nsec): min=2625, max=47760, avg=2836.99, stdev=284.30
    clat (nsec): min=311, max=110006, avg=9445.45, stdev=1220.52
     lat (usec): min=9, max=112, avg=12.28, stdev= 1.25
    clat percentiles (nsec):
     |  1.00th=[ 9024],  5.00th=[ 9152], 10.00th=[ 9152], 20.00th=[ 9152],
     | 30.00th=[ 9280], 40.00th=[ 9280], 50.00th=[ 9280], 60.00th=[ 9280],
     | 70.00th=[ 9408], 80.00th=[ 9408], 90.00th=[ 9408], 95.00th=[ 9664],
     | 99.00th=[15168], 99.50th=[15808], 99.90th=[20096], 99.95th=[27264],
     | 99.99th=[56064]
   bw (  KiB/s): min=38539, max=38959, per=100.00%, avg=38780.29, stdev=105.85, samples=14
   iops        : min=77078, max=77918, avg=77560.64, stdev=211.75, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 10=96.86%, 20=3.03%, 50=0.08%, 100=0.02%, 250=0.01%
  cpu          : usr=13.31%, sys=48.01%, ctx=542695, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=542838,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=37.9MiB/s (39.7MB/s), 37.9MiB/s-37.9MiB/s (39.7MB/s-39.7MB/s), io=265MiB (278MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_io_uring-xnvme_2.txt
$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_io_uring-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751225,
  "timestamp_ms" : 1666751225436,
  "time" : "Wed Oct 26 04:27:05 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 275073536,
        "io_kbytes" : 268626,
        "bw_bytes" : 39290606,
        "bw" : 38369,
        "iops" : 76739.465791,
        "runtime" : 7001,
        "total_ios" : 537253,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2655,
          "max" : 33883,
          "mean" : 2914.394468,
          "stddev" : 191.365400,
          "N" : 537253
        },
        "clat_ns" : {
          "min" : 301,
          "max" : 98825,
          "mean" : 9502.469501,
          "stddev" : 1206.508405,
          "N" : 537253,
          "percentile" : {
            "1.000000" : 9024,
            "5.000000" : 9152,
            "10.000000" : 9152,
            "20.000000" : 9280,
            "30.000000" : 9280,
            "40.000000" : 9280,
            "50.000000" : 9408,
            "60.000000" : 9408,
            "70.000000" : 9408,
            "80.000000" : 9408,
            "90.000000" : 9536,
            "95.000000" : 9664,
            "99.000000" : 15168,
            "99.500000" : 15680,
            "99.900000" : 19840,
            "99.950000" : 25472,
            "99.990000" : 57600
          }
        },
        "lat_ns" : {
          "min" : 9358,
          "max" : 101720,
          "mean" : 12416.863969,
          "stddev" : 1220.253756,
          "N" : 537253
        },
        "bw_min" : 38246,
        "bw_max" : 38615,
        "bw_agg" : 100.000000,
        "bw_mean" : 38379.428571,
        "bw_dev" : 100.427329,
        "bw_samples" : 14,
        "iops_min" : 76492,
        "iops_max" : 77230,
        "iops_mean" : 76758.857143,
        "iops_stddev" : 200.854657,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.200000,
      "sys_cpu" : 46.171429,
      "ctx" : 537127,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.011726,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.830916,
        "20" : 3.088675,
        "50" : 0.051372,
        "100" : 0.017310,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=139549: Wed Oct 26 04:27:05 2022
  read: IOPS=76.7k, BW=37.5MiB/s (39.3MB/s)(262MiB/7001msec)
    slat (nsec): min=2655, max=33883, avg=2914.39, stdev=191.37
    clat (nsec): min=301, max=98825, avg=9502.47, stdev=1206.51
     lat (usec): min=9, max=101, avg=12.42, stdev= 1.22
    clat percentiles (nsec):
     |  1.00th=[ 9024],  5.00th=[ 9152], 10.00th=[ 9152], 20.00th=[ 9280],
     | 30.00th=[ 9280], 40.00th=[ 9280], 50.00th=[ 9408], 60.00th=[ 9408],
     | 70.00th=[ 9408], 80.00th=[ 9408], 90.00th=[ 9536], 95.00th=[ 9664],
     | 99.00th=[15168], 99.50th=[15680], 99.90th=[19840], 99.95th=[25472],
     | 99.99th=[57600]
   bw (  KiB/s): min=38246, max=38615, per=100.00%, avg=38379.43, stdev=100.43, samples=14
   iops        : min=76492, max=77230, avg=76758.86, stdev=200.85, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=96.83%, 20=3.09%, 50=0.05%, 100=0.02%
  cpu          : usr=15.20%, sys=46.17%, ctx=537127, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=537253,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=37.5MiB/s (39.3MB/s), 37.5MiB/s-37.5MiB/s (39.3MB/s-39.3MB/s), io=262MiB (275MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-reference_0.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751239,
  "timestamp_ms" : 1666751239082,
  "time" : "Wed Oct 26 04:27:19 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 285667328,
        "io_kbytes" : 278972,
        "bw_bytes" : 40803789,
        "bw" : 39847,
        "iops" : 79694.900728,
        "runtime" : 7001,
        "total_ios" : 557944,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2726,
          "max" : 35376,
          "mean" : 2949.894054,
          "stddev" : 248.175186,
          "N" : 557944
        },
        "clat_ns" : {
          "min" : 801,
          "max" : 72345,
          "mean" : 8997.065191,
          "stddev" : 1285.566522,
          "N" : 557944,
          "percentile" : {
            "1.000000" : 8640,
            "5.000000" : 8640,
            "10.000000" : 8640,
            "20.000000" : 8768,
            "30.000000" : 8768,
            "40.000000" : 8768,
            "50.000000" : 8768,
            "60.000000" : 8896,
            "70.000000" : 8896,
            "80.000000" : 8896,
            "90.000000" : 9024,
            "95.000000" : 9152,
            "99.000000" : 15168,
            "99.500000" : 15808,
            "99.900000" : 19328,
            "99.950000" : 26240,
            "99.990000" : 55552
          }
        },
        "lat_ns" : {
          "min" : 9508,
          "max" : 75291,
          "mean" : 11946.959245,
          "stddev" : 1309.323831,
          "N" : 557944
        },
        "bw_min" : 39689,
        "bw_max" : 39936,
        "bw_agg" : 100.000000,
        "bw_mean" : 39863.642857,
        "bw_dev" : 67.919191,
        "bw_samples" : 14,
        "iops_min" : 79378,
        "iops_max" : 79872,
        "iops_mean" : 79727.357143,
        "iops_stddev" : 135.822060,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.842857,
      "sys_cpu" : 45.585714,
      "ctx" : 557800,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 97.375543,
        "20" : 2.532512,
        "50" : 0.062013,
        "100" : 0.019536,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 799685,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5026,
      "write_ticks" : 0,
      "in_queue" : 5026,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=139589: Wed Oct 26 04:27:19 2022
  read: IOPS=79.7k, BW=38.9MiB/s (40.8MB/s)(272MiB/7001msec)
    slat (nsec): min=2726, max=35376, avg=2949.89, stdev=248.18
    clat (nsec): min=801, max=72345, avg=8997.07, stdev=1285.57
     lat (nsec): min=9508, max=75291, avg=11946.96, stdev=1309.32
    clat percentiles (nsec):
     |  1.00th=[ 8640],  5.00th=[ 8640], 10.00th=[ 8640], 20.00th=[ 8768],
     | 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8896],
     | 70.00th=[ 8896], 80.00th=[ 8896], 90.00th=[ 9024], 95.00th=[ 9152],
     | 99.00th=[15168], 99.50th=[15808], 99.90th=[19328], 99.95th=[26240],
     | 99.99th=[55552]
   bw (  KiB/s): min=39689, max=39936, per=100.00%, avg=39863.64, stdev=67.92, samples=14
   iops        : min=79378, max=79872, avg=79727.36, stdev=135.82, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=97.38%, 20=2.53%, 50=0.06%, 100=0.02%
  cpu          : usr=14.84%, sys=45.59%, ctx=557800, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=557944,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=38.9MiB/s (40.8MB/s), 38.9MiB/s-38.9MiB/s (40.8MB/s-40.8MB/s), io=272MiB (286MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=799685/0, merge=0/0, ticks=5026/0, in_queue=5026, util=99.09%

$ rm /tmp/fio-output_512_1_libaio-reference_1.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751252,
  "timestamp_ms" : 1666751252703,
  "time" : "Wed Oct 26 04:27:32 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 435968000,
        "io_kbytes" : 425750,
        "bw_bytes" : 62272246,
        "bw" : 60812,
        "iops" : 121625.482074,
        "runtime" : 7001,
        "total_ios" : 851500,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1543,
          "max" : 14407,
          "mean" : 1658.127843,
          "stddev" : 52.610402,
          "N" : 851500
        },
        "clat_ns" : {
          "min" : 410,
          "max" : 96491,
          "mean" : 6259.916447,
          "stddev" : 1052.788853,
          "N" : 851500,
          "percentile" : {
            "1.000000" : 5920,
            "5.000000" : 5984,
            "10.000000" : 5984,
            "20.000000" : 6048,
            "30.000000" : 6048,
            "40.000000" : 6112,
            "50.000000" : 6112,
            "60.000000" : 6176,
            "70.000000" : 6176,
            "80.000000" : 6240,
            "90.000000" : 6368,
            "95.000000" : 6432,
            "99.000000" : 11968,
            "99.500000" : 12480,
            "99.900000" : 15552,
            "99.950000" : 15680,
            "99.990000" : 56064
          }
        },
        "lat_ns" : {
          "min" : 6091,
          "max" : 98144,
          "mean" : 7918.044290,
          "stddev" : 1054.920677,
          "N" : 851500
        },
        "bw_min" : 60701,
        "bw_max" : 61033,
        "bw_agg" : 100.000000,
        "bw_mean" : 60838.642857,
        "bw_dev" : 110.521566,
        "bw_samples" : 14,
        "iops_min" : 121402,
        "iops_max" : 122066,
        "iops_mean" : 121677.428571,
        "iops_stddev" : 221.098208,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.700000,
      "sys_cpu" : 36.571429,
      "ctx" : 851445,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 98.463065,
        "20" : 1.515326,
        "50" : 0.010000,
        "100" : 0.011861,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1221975,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5983,
      "write_ticks" : 0,
      "in_queue" : 5983,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=139627: Wed Oct 26 04:27:32 2022
  read: IOPS=122k, BW=59.4MiB/s (62.3MB/s)(416MiB/7001msec)
    slat (nsec): min=1543, max=14407, avg=1658.13, stdev=52.61
    clat (nsec): min=410, max=96491, avg=6259.92, stdev=1052.79
     lat (nsec): min=6091, max=98144, avg=7918.04, stdev=1054.92
    clat percentiles (nsec):
     |  1.00th=[ 5920],  5.00th=[ 5984], 10.00th=[ 5984], 20.00th=[ 6048],
     | 30.00th=[ 6048], 40.00th=[ 6112], 50.00th=[ 6112], 60.00th=[ 6176],
     | 70.00th=[ 6176], 80.00th=[ 6240], 90.00th=[ 6368], 95.00th=[ 6432],
     | 99.00th=[11968], 99.50th=[12480], 99.90th=[15552], 99.95th=[15680],
     | 99.99th=[56064]
   bw (  KiB/s): min=60701, max=61033, per=100.00%, avg=60838.64, stdev=110.52, samples=14
   iops        : min=121402, max=122066, avg=121677.43, stdev=221.10, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=98.46%, 20=1.52%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=13.70%, sys=36.57%, ctx=851445, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=851500,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=59.4MiB/s (62.3MB/s), 59.4MiB/s-59.4MiB/s (62.3MB/s-62.3MB/s), io=416MiB (436MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1221975/0, merge=0/0, ticks=5983/0, in_queue=5983, util=99.09%

$ rm /tmp/fio-output_512_1_libaio-reference_2.txt
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-reference_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751266,
  "timestamp_ms" : 1666751266367,
  "time" : "Wed Oct 26 04:27:46 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 281868800,
        "io_kbytes" : 275262,
        "bw_bytes" : 40261219,
        "bw" : 39317,
        "iops" : 78635.194972,
        "runtime" : 7001,
        "total_ios" : 550525,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2765,
          "max" : 34235,
          "mean" : 2952.005994,
          "stddev" : 196.975536,
          "N" : 550525
        },
        "clat_ns" : {
          "min" : 791,
          "max" : 76494,
          "mean" : 9169.578695,
          "stddev" : 1235.513611,
          "N" : 550525,
          "percentile" : {
            "1.000000" : 8768,
            "5.000000" : 8768,
            "10.000000" : 8896,
            "20.000000" : 8896,
            "30.000000" : 8896,
            "40.000000" : 9024,
            "50.000000" : 9024,
            "60.000000" : 9024,
            "70.000000" : 9024,
            "80.000000" : 9152,
            "90.000000" : 9152,
            "95.000000" : 9280,
            "99.000000" : 15040,
            "99.500000" : 15680,
            "99.900000" : 19328,
            "99.950000" : 23936,
            "99.990000" : 57600
          }
        },
        "lat_ns" : {
          "min" : 9348,
          "max" : 79429,
          "mean" : 12121.584689,
          "stddev" : 1250.717136,
          "N" : 550525
        },
        "bw_min" : 39172,
        "bw_max" : 39524,
        "bw_agg" : 100.000000,
        "bw_mean" : 39335.142857,
        "bw_dev" : 102.275538,
        "bw_samples" : 14,
        "iops_min" : 78344,
        "iops_max" : 79048,
        "iops_mean" : 78670.428571,
        "iops_stddev" : 204.625630,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.157143,
      "sys_cpu" : 44.100000,
      "ctx" : 550395,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 97.186867,
        "20" : 2.744653,
        "50" : 0.040507,
        "100" : 0.018709,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 790291,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5056,
      "write_ticks" : 0,
      "in_queue" : 5056,
      "util" : 99.058940
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=139665: Wed Oct 26 04:27:46 2022
  read: IOPS=78.6k, BW=38.4MiB/s (40.3MB/s)(269MiB/7001msec)
    slat (nsec): min=2765, max=34235, avg=2952.01, stdev=196.98
    clat (nsec): min=791, max=76494, avg=9169.58, stdev=1235.51
     lat (nsec): min=9348, max=79429, avg=12121.58, stdev=1250.72
    clat percentiles (nsec):
     |  1.00th=[ 8768],  5.00th=[ 8768], 10.00th=[ 8896], 20.00th=[ 8896],
     | 30.00th=[ 8896], 40.00th=[ 9024], 50.00th=[ 9024], 60.00th=[ 9024],
     | 70.00th=[ 9024], 80.00th=[ 9152], 90.00th=[ 9152], 95.00th=[ 9280],
     | 99.00th=[15040], 99.50th=[15680], 99.90th=[19328], 99.95th=[23936],
     | 99.99th=[57600]
   bw (  KiB/s): min=39172, max=39524, per=100.00%, avg=39335.14, stdev=102.28, samples=14
   iops        : min=78344, max=79048, avg=78670.43, stdev=204.63, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=97.19%, 20=2.74%, 50=0.04%, 100=0.02%
  cpu          : usr=16.16%, sys=44.10%, ctx=550395, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=550525,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=38.4MiB/s (40.3MB/s), 38.4MiB/s-38.4MiB/s (40.3MB/s-40.3MB/s), io=269MiB (282MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=790291/0, merge=0/0, ticks=5056/0, in_queue=5056, util=99.06%

$ rm /tmp/fio-output_512_1_libaio-bdev_aio_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751281,
  "timestamp_ms" : 1666751281368,
  "time" : "Wed Oct 26 04:28:01 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 568576000,
        "io_kbytes" : 555250,
        "bw_bytes" : 81225142,
        "bw" : 79321,
        "iops" : 158642.857143,
        "runtime" : 7000,
        "total_ios" : 1110500,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1613,
          "max" : 37350,
          "mean" : 1678.867968,
          "stddev" : 146.532978,
          "N" : 1110500
        },
        "clat_ns" : {
          "min" : 190,
          "max" : 115837,
          "mean" : 4397.280773,
          "stddev" : 1142.935701,
          "N" : 1110500,
          "percentile" : {
            "1.000000" : 4128,
            "5.000000" : 4128,
            "10.000000" : 4128,
            "20.000000" : 4192,
            "30.000000" : 4192,
            "40.000000" : 4192,
            "50.000000" : 4256,
            "60.000000" : 4256,
            "70.000000" : 4320,
            "80.000000" : 4320,
            "90.000000" : 4384,
            "95.000000" : 4576,
            "99.000000" : 9792,
            "99.500000" : 11328,
            "99.900000" : 15552,
            "99.950000" : 15680,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5730,
          "max" : 117510,
          "mean" : 6076.148740,
          "stddev" : 1154.189344,
          "N" : 1110500
        },
        "bw_min" : 79120,
        "bw_max" : 79623,
        "bw_agg" : 100.000000,
        "bw_mean" : 79328.214286,
        "bw_dev" : 158.211241,
        "bw_samples" : 14,
        "iops_min" : 158240,
        "iops_max" : 159246,
        "iops_mean" : 158656.500000,
        "iops_stddev" : 316.477913,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 76.614286,
      "sys_cpu" : 23.357143,
      "ctx" : 26,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 99.038451,
        "20" : 0.941108,
        "50" : 0.010000,
        "100" : 0.012517,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=139713: Wed Oct 26 04:28:01 2022
  read: IOPS=159k, BW=77.5MiB/s (81.2MB/s)(542MiB/7000msec)
    slat (nsec): min=1613, max=37350, avg=1678.87, stdev=146.53
    clat (nsec): min=190, max=115837, avg=4397.28, stdev=1142.94
     lat (usec): min=5, max=117, avg= 6.08, stdev= 1.15
    clat percentiles (nsec):
     |  1.00th=[ 4128],  5.00th=[ 4128], 10.00th=[ 4128], 20.00th=[ 4192],
     | 30.00th=[ 4192], 40.00th=[ 4192], 50.00th=[ 4256], 60.00th=[ 4256],
     | 70.00th=[ 4320], 80.00th=[ 4320], 90.00th=[ 4384], 95.00th=[ 4576],
     | 99.00th=[ 9792], 99.50th=[11328], 99.90th=[15552], 99.95th=[15680],
     | 99.99th=[57088]
   bw (  KiB/s): min=79120, max=79623, per=100.00%, avg=79328.21, stdev=158.21, samples=14
   iops        : min=158240, max=159246, avg=158656.50, stdev=316.48, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=99.04%, 20=0.94%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=76.61%, sys=23.36%, ctx=26, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1110500,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=77.5MiB/s (81.2MB/s), 77.5MiB/s-77.5MiB/s (81.2MB/s-81.2MB/s), io=542MiB (569MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_libaio-bdev_aio_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751296,
  "timestamp_ms" : 1666751296479,
  "time" : "Wed Oct 26 04:28:16 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 569852416,
        "io_kbytes" : 556496,
        "bw_bytes" : 81407488,
        "bw" : 79499,
        "iops" : 158999.000000,
        "runtime" : 7000,
        "total_ios" : 1112993,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1623,
          "max" : 25918,
          "mean" : 1684.411642,
          "stddev" : 128.915050,
          "N" : 1112993
        },
        "clat_ns" : {
          "min" : 210,
          "max" : 76463,
          "mean" : 4380.684294,
          "stddev" : 1145.616557,
          "N" : 1112993,
          "percentile" : {
            "1.000000" : 4128,
            "5.000000" : 4128,
            "10.000000" : 4128,
            "20.000000" : 4128,
            "30.000000" : 4192,
            "40.000000" : 4192,
            "50.000000" : 4256,
            "60.000000" : 4256,
            "70.000000" : 4256,
            "80.000000" : 4320,
            "90.000000" : 4384,
            "95.000000" : 4512,
            "99.000000" : 9792,
            "99.500000" : 11200,
            "99.900000" : 15424,
            "99.950000" : 15680,
            "99.990000" : 57600
          }
        },
        "lat_ns" : {
          "min" : 5731,
          "max" : 78166,
          "mean" : 6065.095936,
          "stddev" : 1154.036926,
          "N" : 1112993
        },
        "bw_min" : 79322,
        "bw_max" : 79715,
        "bw_agg" : 100.000000,
        "bw_mean" : 79509.214286,
        "bw_dev" : 119.767196,
        "bw_samples" : 14,
        "iops_min" : 158644,
        "iops_max" : 159430,
        "iops_mean" : 159018.571429,
        "iops_stddev" : 239.596364,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 76.442857,
      "sys_cpu" : 23.528571,
      "ctx" : 26,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 99.035214,
        "20" : 0.943762,
        "50" : 0.010000,
        "100" : 0.013208,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=139762: Wed Oct 26 04:28:16 2022
  read: IOPS=159k, BW=77.6MiB/s (81.4MB/s)(543MiB/7000msec)
    slat (nsec): min=1623, max=25918, avg=1684.41, stdev=128.92
    clat (nsec): min=210, max=76463, avg=4380.68, stdev=1145.62
     lat (nsec): min=5731, max=78166, avg=6065.10, stdev=1154.04
    clat percentiles (nsec):
     |  1.00th=[ 4128],  5.00th=[ 4128], 10.00th=[ 4128], 20.00th=[ 4128],
     | 30.00th=[ 4192], 40.00th=[ 4192], 50.00th=[ 4256], 60.00th=[ 4256],
     | 70.00th=[ 4256], 80.00th=[ 4320], 90.00th=[ 4384], 95.00th=[ 4512],
     | 99.00th=[ 9792], 99.50th=[11200], 99.90th=[15424], 99.95th=[15680],
     | 99.99th=[57600]
   bw (  KiB/s): min=79322, max=79715, per=100.00%, avg=79509.21, stdev=119.77, samples=14
   iops        : min=158644, max=159430, avg=159018.57, stdev=239.60, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=99.04%, 20=0.94%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=76.44%, sys=23.53%, ctx=26, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1112993,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=77.6MiB/s (81.4MB/s), 77.6MiB/s-77.6MiB/s (81.4MB/s-81.4MB/s), io=543MiB (570MB), run=7000-7000msec

$ rm /tmp/fio-output_512_1_libaio-bdev_aio_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_aio_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751311,
  "timestamp_ms" : 1666751311692,
  "time" : "Wed Oct 26 04:28:31 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 570132480,
        "io_kbytes" : 556770,
        "bw_bytes" : 81435863,
        "bw" : 79527,
        "iops" : 159054.277960,
        "runtime" : 7001,
        "total_ios" : 1113539,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1603,
          "max" : 60874,
          "mean" : 1654.208657,
          "stddev" : 140.431430,
          "N" : 1113540
        },
        "clat_ns" : {
          "min" : 200,
          "max" : 114164,
          "mean" : 4398.236214,
          "stddev" : 1110.230874,
          "N" : 1113539,
          "percentile" : {
            "1.000000" : 4128,
            "5.000000" : 4128,
            "10.000000" : 4192,
            "20.000000" : 4192,
            "30.000000" : 4192,
            "40.000000" : 4192,
            "50.000000" : 4256,
            "60.000000" : 4256,
            "70.000000" : 4320,
            "80.000000" : 4320,
            "90.000000" : 4384,
            "95.000000" : 4512,
            "99.000000" : 9920,
            "99.500000" : 10560,
            "99.900000" : 15552,
            "99.950000" : 15680,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5730,
          "max" : 115827,
          "mean" : 6052.444891,
          "stddev" : 1120.267316,
          "N" : 1113539
        },
        "bw_min" : 79426,
        "bw_max" : 79846,
        "bw_agg" : 100.000000,
        "bw_mean" : 79546.428571,
        "bw_dev" : 125.765175,
        "bw_samples" : 14,
        "iops_min" : 158852,
        "iops_max" : 159693,
        "iops_mean" : 159092.785714,
        "iops_stddev" : 251.830889,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 76.100000,
      "sys_cpu" : 23.871429,
      "ctx" : 25,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 99.023653,
        "20" : 0.955692,
        "50" : 0.010000,
        "100" : 0.011674,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=139811: Wed Oct 26 04:28:31 2022
  read: IOPS=159k, BW=77.7MiB/s (81.4MB/s)(544MiB/7001msec)
    slat (nsec): min=1603, max=60874, avg=1654.21, stdev=140.43
    clat (nsec): min=200, max=114164, avg=4398.24, stdev=1110.23
     lat (usec): min=5, max=115, avg= 6.05, stdev= 1.12
    clat percentiles (nsec):
     |  1.00th=[ 4128],  5.00th=[ 4128], 10.00th=[ 4192], 20.00th=[ 4192],
     | 30.00th=[ 4192], 40.00th=[ 4192], 50.00th=[ 4256], 60.00th=[ 4256],
     | 70.00th=[ 4320], 80.00th=[ 4320], 90.00th=[ 4384], 95.00th=[ 4512],
     | 99.00th=[ 9920], 99.50th=[10560], 99.90th=[15552], 99.95th=[15680],
     | 99.99th=[57088]
   bw (  KiB/s): min=79426, max=79846, per=100.00%, avg=79546.43, stdev=125.77, samples=14
   iops        : min=158852, max=159693, avg=159092.79, stdev=251.83, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=99.02%, 20=0.96%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=76.10%, sys=23.87%, ctx=25, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1113539,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=77.7MiB/s (81.4MB/s), 77.7MiB/s-77.7MiB/s (81.4MB/s-81.4MB/s), io=544MiB (570MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751326,
  "timestamp_ms" : 1666751326817,
  "time" : "Wed Oct 26 04:28:46 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 549307392,
        "io_kbytes" : 536433,
        "bw_bytes" : 78461275,
        "bw" : 76622,
        "iops" : 153244.679332,
        "runtime" : 7001,
        "total_ios" : 1072866,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1623,
          "max" : 14217,
          "mean" : 1678.038774,
          "stddev" : 119.906182,
          "N" : 1072866
        },
        "clat_ns" : {
          "min" : 591,
          "max" : 118713,
          "mean" : 4583.107354,
          "stddev" : 1160.723079,
          "N" : 1072866,
          "percentile" : {
            "1.000000" : 4320,
            "5.000000" : 4320,
            "10.000000" : 4320,
            "20.000000" : 4320,
            "30.000000" : 4320,
            "40.000000" : 4320,
            "50.000000" : 4384,
            "60.000000" : 4384,
            "70.000000" : 4640,
            "80.000000" : 4640,
            "90.000000" : 4704,
            "95.000000" : 4768,
            "99.000000" : 9792,
            "99.500000" : 10816,
            "99.900000" : 15552,
            "99.950000" : 15680,
            "99.990000" : 59136
          }
        },
        "lat_ns" : {
          "min" : 5881,
          "max" : 120406,
          "mean" : 6261.146127,
          "stddev" : 1168.029557,
          "N" : 1072866
        },
        "bw_min" : 76498,
        "bw_max" : 76736,
        "bw_agg" : 100.000000,
        "bw_mean" : 76644.785714,
        "bw_dev" : 66.817983,
        "bw_samples" : 14,
        "iops_min" : 152997,
        "iops_max" : 153472,
        "iops_mean" : 153289.642857,
        "iops_stddev" : 133.467142,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 36.614286,
      "sys_cpu" : 63.357143,
      "ctx" : 23,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 99.073137,
        "20" : 0.903188,
        "50" : 0.010000,
        "100" : 0.015379,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139861: Wed Oct 26 04:28:46 2022
  read: IOPS=153k, BW=74.8MiB/s (78.5MB/s)(524MiB/7001msec)
    slat (nsec): min=1623, max=14217, avg=1678.04, stdev=119.91
    clat (nsec): min=591, max=118713, avg=4583.11, stdev=1160.72
     lat (usec): min=5, max=120, avg= 6.26, stdev= 1.17
    clat percentiles (nsec):
     |  1.00th=[ 4320],  5.00th=[ 4320], 10.00th=[ 4320], 20.00th=[ 4320],
     | 30.00th=[ 4320], 40.00th=[ 4320], 50.00th=[ 4384], 60.00th=[ 4384],
     | 70.00th=[ 4640], 80.00th=[ 4640], 90.00th=[ 4704], 95.00th=[ 4768],
     | 99.00th=[ 9792], 99.50th=[10816], 99.90th=[15552], 99.95th=[15680],
     | 99.99th=[59136]
   bw (  KiB/s): min=76498, max=76736, per=100.00%, avg=76644.79, stdev=66.82, samples=14
   iops        : min=152997, max=153472, avg=153289.64, stdev=133.47, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=99.07%, 20=0.90%, 50=0.01%
  lat (usec)   : 100=0.02%, 250=0.01%
  cpu          : usr=36.61%, sys=63.36%, ctx=23, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1072866,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=74.8MiB/s (78.5MB/s), 74.8MiB/s-74.8MiB/s (78.5MB/s-78.5MB/s), io=524MiB (549MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751341,
  "timestamp_ms" : 1666751341933,
  "time" : "Wed Oct 26 04:29:01 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 548251136,
        "io_kbytes" : 535401,
        "bw_bytes" : 78310403,
        "bw" : 76475,
        "iops" : 152950.007142,
        "runtime" : 7001,
        "total_ios" : 1070803,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1633,
          "max" : 14327,
          "mean" : 1697.999735,
          "stddev" : 113.979161,
          "N" : 1070803
        },
        "clat_ns" : {
          "min" : 602,
          "max" : 97693,
          "mean" : 4606.212420,
          "stddev" : 1101.921202,
          "N" : 1070803,
          "percentile" : {
            "1.000000" : 4384,
            "5.000000" : 4384,
            "10.000000" : 4384,
            "20.000000" : 4384,
            "30.000000" : 4384,
            "40.000000" : 4448,
            "50.000000" : 4448,
            "60.000000" : 4448,
            "70.000000" : 4448,
            "80.000000" : 4512,
            "90.000000" : 4704,
            "95.000000" : 4768,
            "99.000000" : 9792,
            "99.500000" : 10816,
            "99.900000" : 15552,
            "99.950000" : 15680,
            "99.990000" : 57088
          }
        },
        "lat_ns" : {
          "min" : 5921,
          "max" : 99376,
          "mean" : 6304.212155,
          "stddev" : 1109.513640,
          "N" : 1070803
        },
        "bw_min" : 76350,
        "bw_max" : 76686,
        "bw_agg" : 100.000000,
        "bw_mean" : 76496.357143,
        "bw_dev" : 105.615855,
        "bw_samples" : 14,
        "iops_min" : 152700,
        "iops_max" : 153372,
        "iops_mean" : 152992.928571,
        "iops_stddev" : 211.019673,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 29.985714,
      "sys_cpu" : 69.985714,
      "ctx" : 20,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 99.040907,
        "20" : 0.937054,
        "50" : 0.010000,
        "100" : 0.013168,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139913: Wed Oct 26 04:29:01 2022
  read: IOPS=153k, BW=74.7MiB/s (78.3MB/s)(523MiB/7001msec)
    slat (nsec): min=1633, max=14327, avg=1698.00, stdev=113.98
    clat (nsec): min=602, max=97693, avg=4606.21, stdev=1101.92
     lat (nsec): min=5921, max=99376, avg=6304.21, stdev=1109.51
    clat percentiles (nsec):
     |  1.00th=[ 4384],  5.00th=[ 4384], 10.00th=[ 4384], 20.00th=[ 4384],
     | 30.00th=[ 4384], 40.00th=[ 4448], 50.00th=[ 4448], 60.00th=[ 4448],
     | 70.00th=[ 4448], 80.00th=[ 4512], 90.00th=[ 4704], 95.00th=[ 4768],
     | 99.00th=[ 9792], 99.50th=[10816], 99.90th=[15552], 99.95th=[15680],
     | 99.99th=[57088]
   bw (  KiB/s): min=76350, max=76686, per=100.00%, avg=76496.36, stdev=105.62, samples=14
   iops        : min=152700, max=153372, avg=152992.93, stdev=211.02, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=99.04%, 20=0.94%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=29.99%, sys=69.99%, ctx=20, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1070803,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=74.7MiB/s (78.3MB/s), 74.7MiB/s-74.7MiB/s (78.3MB/s-78.3MB/s), io=523MiB (548MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt" --cpus_allowed="1"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_512_1_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751357,
  "timestamp_ms" : 1666751357029,
  "time" : "Wed Oct 26 04:29:17 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 552497664,
        "io_kbytes" : 539548,
        "bw_bytes" : 78916963,
        "bw" : 77067,
        "iops" : 154134.695044,
        "runtime" : 7001,
        "total_ios" : 1079097,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1613,
          "max" : 25488,
          "mean" : 1684.209908,
          "stddev" : 120.441497,
          "N" : 1079097
        },
        "clat_ns" : {
          "min" : 592,
          "max" : 76343,
          "mean" : 4568.302466,
          "stddev" : 1090.399727,
          "N" : 1079097,
          "percentile" : {
            "1.000000" : 4320,
            "5.000000" : 4320,
            "10.000000" : 4320,
            "20.000000" : 4384,
            "30.000000" : 4384,
            "40.000000" : 4384,
            "50.000000" : 4384,
            "60.000000" : 4448,
            "70.000000" : 4448,
            "80.000000" : 4448,
            "90.000000" : 4704,
            "95.000000" : 4768,
            "99.000000" : 9792,
            "99.500000" : 11328,
            "99.900000" : 15552,
            "99.950000" : 15680,
            "99.990000" : 56576
          }
        },
        "lat_ns" : {
          "min" : 5861,
          "max" : 78016,
          "mean" : 6252.512374,
          "stddev" : 1098.558092,
          "N" : 1079097
        },
        "bw_min" : 76780,
        "bw_max" : 77307,
        "bw_agg" : 100.000000,
        "bw_mean" : 77088.357143,
        "bw_dev" : 198.919236,
        "bw_samples" : 14,
        "iops_min" : 153560,
        "iops_max" : 154614,
        "iops_mean" : 154176.714286,
        "iops_stddev" : 397.838473,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 29.242857,
      "sys_cpu" : 70.728571,
      "ctx" : 22,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 99.054580,
        "20" : 0.927349,
        "50" : 0.010000,
        "100" : 0.010842,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=139972: Wed Oct 26 04:29:17 2022
  read: IOPS=154k, BW=75.3MiB/s (78.9MB/s)(527MiB/7001msec)
    slat (nsec): min=1613, max=25488, avg=1684.21, stdev=120.44
    clat (nsec): min=592, max=76343, avg=4568.30, stdev=1090.40
     lat (nsec): min=5861, max=78016, avg=6252.51, stdev=1098.56
    clat percentiles (nsec):
     |  1.00th=[ 4320],  5.00th=[ 4320], 10.00th=[ 4320], 20.00th=[ 4384],
     | 30.00th=[ 4384], 40.00th=[ 4384], 50.00th=[ 4384], 60.00th=[ 4448],
     | 70.00th=[ 4448], 80.00th=[ 4448], 90.00th=[ 4704], 95.00th=[ 4768],
     | 99.00th=[ 9792], 99.50th=[11328], 99.90th=[15552], 99.95th=[15680],
     | 99.99th=[56576]
   bw (  KiB/s): min=76780, max=77307, per=100.00%, avg=77088.36, stdev=198.92, samples=14
   iops        : min=153560, max=154614, avg=154176.71, stdev=397.84, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=99.05%, 20=0.93%, 50=0.01%
  lat (usec)   : 100=0.01%
  cpu          : usr=29.24%, sys=70.73%, ctx=22, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1079097,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=75.3MiB/s (78.9MB/s), 75.3MiB/s-75.3MiB/s (78.9MB/s-78.9MB/s), io=527MiB (552MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-xnvme_0.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_0.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751370,
  "timestamp_ms" : 1666751370821,
  "time" : "Wed Oct 26 04:29:30 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 263074816,
        "io_kbytes" : 256909,
        "bw_bytes" : 37576748,
        "bw" : 36696,
        "iops" : 73392.086845,
        "runtime" : 7001,
        "total_ios" : 513818,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2856,
          "max" : 24596,
          "mean" : 3174.835060,
          "stddev" : 192.695012,
          "N" : 513818
        },
        "clat_ns" : {
          "min" : 942,
          "max" : 70743,
          "mean" : 9889.292977,
          "stddev" : 1172.701180,
          "N" : 513818,
          "percentile" : {
            "1.000000" : 8512,
            "5.000000" : 9536,
            "10.000000" : 9664,
            "20.000000" : 9664,
            "30.000000" : 9792,
            "40.000000" : 9792,
            "50.000000" : 9792,
            "60.000000" : 9792,
            "70.000000" : 9792,
            "80.000000" : 9792,
            "90.000000" : 9920,
            "95.000000" : 10048,
            "99.000000" : 15040,
            "99.500000" : 16192,
            "99.900000" : 20096,
            "99.950000" : 24960,
            "99.990000" : 58112
          }
        },
        "lat_ns" : {
          "min" : 8586,
          "max" : 73798,
          "mean" : 13064.128038,
          "stddev" : 1183.145219,
          "N" : 513818
        },
        "bw_min" : 36599,
        "bw_max" : 36839,
        "bw_agg" : 100.000000,
        "bw_mean" : 36707.785714,
        "bw_dev" : 78.708985,
        "bw_samples" : 14,
        "iops_min" : 73198,
        "iops_max" : 73678,
        "iops_mean" : 73415.428571,
        "iops_stddev" : 157.295271,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.328571,
      "sys_cpu" : 47.942857,
      "ctx" : 513642,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010120,
        "4" : 0.000000,
        "10" : 94.480731,
        "20" : 5.399772,
        "50" : 0.087969,
        "100" : 0.019657,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=140012: Wed Oct 26 04:29:30 2022
  read: IOPS=73.4k, BW=35.8MiB/s (37.6MB/s)(251MiB/7001msec)
    slat (nsec): min=2856, max=24596, avg=3174.84, stdev=192.70
    clat (nsec): min=942, max=70743, avg=9889.29, stdev=1172.70
     lat (nsec): min=8586, max=73798, avg=13064.13, stdev=1183.15
    clat percentiles (nsec):
     |  1.00th=[ 8512],  5.00th=[ 9536], 10.00th=[ 9664], 20.00th=[ 9664],
     | 30.00th=[ 9792], 40.00th=[ 9792], 50.00th=[ 9792], 60.00th=[ 9792],
     | 70.00th=[ 9792], 80.00th=[ 9792], 90.00th=[ 9920], 95.00th=[10048],
     | 99.00th=[15040], 99.50th=[16192], 99.90th=[20096], 99.95th=[24960],
     | 99.99th=[58112]
   bw (  KiB/s): min=36599, max=36839, per=100.00%, avg=36707.79, stdev=78.71, samples=14
   iops        : min=73198, max=73678, avg=73415.43, stdev=157.30, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=94.48%, 20=5.40%, 50=0.09%, 100=0.02%
  cpu          : usr=15.33%, sys=47.94%, ctx=513642, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=513818,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=35.8MiB/s (37.6MB/s), 35.8MiB/s-35.8MiB/s (37.6MB/s-37.6MB/s), io=251MiB (263MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-xnvme_1.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_1.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751384,
  "timestamp_ms" : 1666751384484,
  "time" : "Wed Oct 26 04:29:44 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 345976320,
        "io_kbytes" : 337867,
        "bw_bytes" : 49418128,
        "bw" : 48259,
        "iops" : 96519.782888,
        "runtime" : 7001,
        "total_ios" : 675735,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2174,
          "max" : 24436,
          "mean" : 2431.272401,
          "stddev" : 151.964015,
          "N" : 675735
        },
        "clat_ns" : {
          "min" : 701,
          "max" : 75923,
          "mean" : 7477.687172,
          "stddev" : 1131.118569,
          "N" : 675735,
          "percentile" : {
            "1.000000" : 7136,
            "5.000000" : 7200,
            "10.000000" : 7200,
            "20.000000" : 7264,
            "30.000000" : 7264,
            "40.000000" : 7264,
            "50.000000" : 7328,
            "60.000000" : 7328,
            "70.000000" : 7392,
            "80.000000" : 7392,
            "90.000000" : 7584,
            "95.000000" : 7712,
            "99.000000" : 14016,
            "99.500000" : 15040,
            "99.900000" : 16064,
            "99.950000" : 17024,
            "99.990000" : 56064
          }
        },
        "lat_ns" : {
          "min" : 7504,
          "max" : 78317,
          "mean" : 9908.959573,
          "stddev" : 1141.464837,
          "N" : 675735
        },
        "bw_min" : 48137,
        "bw_max" : 48485,
        "bw_agg" : 100.000000,
        "bw_mean" : 48272.857143,
        "bw_dev" : 96.448358,
        "bw_samples" : 14,
        "iops_min" : 96274,
        "iops_max" : 96970,
        "iops_mean" : 96545.785714,
        "iops_stddev" : 192.890634,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.128571,
      "sys_cpu" : 46.542857,
      "ctx" : 675583,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 98.266776,
        "20" : 1.699483,
        "50" : 0.011839,
        "100" : 0.013023,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=140052: Wed Oct 26 04:29:44 2022
  read: IOPS=96.5k, BW=47.1MiB/s (49.4MB/s)(330MiB/7001msec)
    slat (nsec): min=2174, max=24436, avg=2431.27, stdev=151.96
    clat (nsec): min=701, max=75923, avg=7477.69, stdev=1131.12
     lat (nsec): min=7504, max=78317, avg=9908.96, stdev=1141.46
    clat percentiles (nsec):
     |  1.00th=[ 7136],  5.00th=[ 7200], 10.00th=[ 7200], 20.00th=[ 7264],
     | 30.00th=[ 7264], 40.00th=[ 7264], 50.00th=[ 7328], 60.00th=[ 7328],
     | 70.00th=[ 7392], 80.00th=[ 7392], 90.00th=[ 7584], 95.00th=[ 7712],
     | 99.00th=[14016], 99.50th=[15040], 99.90th=[16064], 99.95th=[17024],
     | 99.99th=[56064]
   bw (  KiB/s): min=48137, max=48485, per=100.00%, avg=48272.86, stdev=96.45, samples=14
   iops        : min=96274, max=96970, avg=96545.79, stdev=192.89, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=98.27%, 20=1.70%, 50=0.01%, 100=0.01%
  cpu          : usr=16.13%, sys=46.54%, ctx=675583, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=675735,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=47.1MiB/s (49.4MB/s), 47.1MiB/s-47.1MiB/s (49.4MB/s-49.4MB/s), io=330MiB (346MB), run=7001-7001msec

$ rm /tmp/fio-output_512_1_libaio-xnvme_2.txt
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="512" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_512_1_libaio-xnvme_2.txt" --cpus_allowed="1"
$ cat /tmp/fio-output_512_1_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666751398,
  "timestamp_ms" : 1666751398140,
  "time" : "Wed Oct 26 04:29:58 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "512",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1",
        "cpus_allowed" : "1"
      },
      "read" : {
        "io_bytes" : 264449536,
        "io_kbytes" : 258251,
        "bw_bytes" : 37773108,
        "bw" : 36887,
        "iops" : 73775.460648,
        "runtime" : 7001,
        "total_ios" : 516502,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2955,
          "max" : 35957,
          "mean" : 3190.098706,
          "stddev" : 227.777047,
          "N" : 516503
        },
        "clat_ns" : {
          "min" : 892,
          "max" : 117179,
          "mean" : 9770.164466,
          "stddev" : 1173.998407,
          "N" : 516502,
          "percentile" : {
            "1.000000" : 8256,
            "5.000000" : 9408,
            "10.000000" : 9536,
            "20.000000" : 9536,
            "30.000000" : 9664,
            "40.000000" : 9664,
            "50.000000" : 9664,
            "60.000000" : 9664,
            "70.000000" : 9664,
            "80.000000" : 9792,
            "90.000000" : 9792,
            "95.000000" : 9920,
            "99.000000" : 14912,
            "99.500000" : 16064,
            "99.900000" : 20096,
            "99.950000" : 25216,
            "99.990000" : 55552
          }
        },
        "lat_ns" : {
          "min" : 9898,
          "max" : 120415,
          "mean" : 12960.263335,
          "stddev" : 1193.805394,
          "N" : 516502
        },
        "bw_min" : 36783,
        "bw_max" : 37012,
        "bw_agg" : 100.000000,
        "bw_mean" : 36898.714286,
        "bw_dev" : 75.942720,
        "bw_samples" : 14,
        "iops_min" : 73566,
        "iops_max" : 74024,
        "iops_mean" : 73797.714286,
        "iops_stddev" : 151.591774,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.985714,
      "sys_cpu" : 47.085714,
      "ctx" : 516350,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 95.703211,
        "20" : 4.180429,
        "50" : 0.089061,
        "100" : 0.017812,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=140092: Wed Oct 26 04:29:58 2022
  read: IOPS=73.8k, BW=36.0MiB/s (37.8MB/s)(252MiB/7001msec)
    slat (nsec): min=2955, max=35957, avg=3190.10, stdev=227.78
    clat (nsec): min=892, max=117179, avg=9770.16, stdev=1174.00
     lat (usec): min=9, max=120, avg=12.96, stdev= 1.19
    clat percentiles (nsec):
     |  1.00th=[ 8256],  5.00th=[ 9408], 10.00th=[ 9536], 20.00th=[ 9536],
     | 30.00th=[ 9664], 40.00th=[ 9664], 50.00th=[ 9664], 60.00th=[ 9664],
     | 70.00th=[ 9664], 80.00th=[ 9792], 90.00th=[ 9792], 95.00th=[ 9920],
     | 99.00th=[14912], 99.50th=[16064], 99.90th=[20096], 99.95th=[25216],
     | 99.99th=[55552]
   bw (  KiB/s): min=36783, max=37012, per=100.00%, avg=36898.71, stdev=75.94, samples=14
   iops        : min=73566, max=74024, avg=73797.71, stdev=151.59, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=95.70%, 20=4.18%, 50=0.09%, 100=0.02%
  lat (usec)   : 250=0.01%
  cpu          : usr=15.99%, sys=47.09%, ctx=516350, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=516502,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=36.0MiB/s (37.8MB/s), 36.0MiB/s-36.0MiB/s (37.8MB/s-37.8MB/s), io=252MiB (264MB), run=7001-7001msec

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Run fio...

{'description': '\n    Run fio...\n',
 'extras': {},
 'id': '004_plot_fio',
 'name': 'plot_fio',
 'nr': 4,
 'status': {'elapsed': 0.6777725219726562,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'plot',
 'with': {'path': 'cijoe-output',
          'plot_attributes': 'plot-attributes-512.yaml',
          'tool': 'fio'}}
EOL: In the presence of failure, recall, this too shall pass.